The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
HFRP
> Out-of-School Time
> OST Database & Bibliography
> Database
> Citizen Schools
> Evaluation #2 (2002–2003): Phase III Findings
A Profile of the Evaluation of Citizen Schools
Profile Information
- Full Text (HTML)
- How to cite
- About the OST Database
Program Description
| Overview | Citizen Schools (CS) operates a national network of apprenticeship programs for middle school students that connects adult volunteers to youth in hands-on afterschool learning projects. The CS programs are designed to help youth develop academic and leadership skills needed to succeed in school, get into college, and become leaders in their careers and in their communities. |
| Start Date | 1995 |
| Scope | national |
| Type | afterschool |
| Location | urban |
| Setting | public school |
| Participants | middle school students (grades 6–8) |
| Number of Sites/Grantees | 30 sites in 2006–2007 nationwide; 37 sites nationwide (2009–2010) |
| Number Served | approximately 3,000 in 2006–2007 nationwide; approximately 4,000 nationwide (2009–2010) |
| Components | Twice a week after school, CS youth participate in apprenticeships, which consist of hands-on learning projects led by volunteer Citizen Teachers. Apprentices work collaboratively in small groups to build solar cars, litigate mock trials, publish children's books, and so on. On other weekday afternoons, CS staff lead structured afterschool activities to enhance school success of youth, working on homework and building organizational and study skills to help youth navigate their education through high school, graduation, and beyond. Each semester culminates in “WOW!”—a public presentation of the CS participants’ projects. In 2001–2002, CS launched its 8th Grade Academy program in Boston for eighth graders who began CS in a prior year. In addition to offering activities similar to those at other CS campuses, 8th Grade Academy is intended to help youth apply to and succeed in competitive high schools and to introduce them to the college application process through experiential learning activities that build academic and life skills and give youth access to coaches, technology, internships, and other educational programs. Each participant is assigned a writing coach (typically a local lawyer). In 2004, CS launched an alumni program to support Academy graduates during the high school transition period. |
| Funding Level | approximately $20.9 million in 2009–2010 |
| Funding Sources | Major funders include ArcLight Capital Partners, The Atlantic Philanthropies, Bank of America, Josh & Anita Bekenstein, Edna McConnell Clark Foundation, The John S. and James L. Knight Foundation, The Koogle Foundation, The Lovett-Woodsum Foundation, The Picower Foundation, The Samberg Family Foundation, Skoll Foundation, AmeriCorps, State of North Carolina Department of Social Servies, Boston Public Schools, Duke Endowment, Houston Independent School District, Nellie Mae Education Foundation, and many other individual, corporate, and philanthropic donors. |
Evaluation
| Overview | A 7-year evaluation (2001–2008) focused on the program experiences and outcomes of 5 cohorts of participants at the Boston site. |
| Evaluators | Policy Studies Associates, Inc. |
| Evaluations Profiled | Evidence from Two Student Cohorts on the Use of Community Resources to Promote Youth Development: Phase II Report Putting Students on a Pathway to Academic and Social Success: Phase III Findings Preparing Students in the Middle Grades to Succeed in High School: Findings from Phase IV Progress Toward High School Graduation: Youth Outcomes in Boston Achieving High School Graduation: Citizen Schools’ Youth Outcomes in Boston |
| Evaluations Planned | Citizen Schools is proceeding with a new evaluation plan in which Abt Associates Inc. and Public/Private Ventures will serve as external evaluators. |
| Report Availability | Fabiano, L., Espino, J., & Reisner, E. R., with Pearson, L. M. (2003). Citizen Schools: Using community resources to promote youth development. Phase I Report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2003.pdf Espino, J., Fabiano, L., & Pearson, L. M. (with Kirkwood K. P., Afolabi, K., & Pasatta, K.). (2004). Citizen Schools: Evidence from two student cohorts on the use of community resources to promote youth development. Phase II report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Fabiano, L., Pearson, L. M., & Williams, I. J. (2005). Putting students on a pathway to academic and social success: Phase III findings of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2005.pdf Fabiano, L., Pearson, L. M., Reisner, E. R., & Williams, I. J. (2006). Preparing students in the middle grades to succeed in high school: Findings from Phase IV of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2006.pdf Pearson, L. M., Vile, J. D., & Reisner, E. R. (2008). Establishing a foundation for progress toward high school graduation: Findings from Phase V of the Citizen Schools Evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2008.pdf Vile, J. D., Arcaira, E., & Reisner, E. R. (2009). Progress toward high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2009.pdf Arcaira, E., Vile, J. D., & Reisner, E. R. (2010). Achieving high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates. Available at: www.policystudies.com/studies/?download=79&id=39 |
Contacts
| Evaluation | Erikson Arcaira Research Associate Policy Studies Associates, Inc. 1718 Connecticut Avenue, NW, Suite 400 Washington, DC 20009 Tel: 202-939-5343 Fax: 202-939-5732 Email: earcaira@policystudies.com |
|
| Program | Michael Kubiak Director of Research and Evaluation Citizen Schools 308 Congress Street, 5th Floor Boston, MA 02210 Tel: 617-695-2300 ext. 132 Fax: 617-695-2367 Email: michaelkubiak@citizenschools.org |
|
| Profile Updated | May 9, 2011 | |
Evaluation 2: Putting Students on a Pathway to Academic and Social Success: Phase III Findings of the Citizen Schools Evaluation
Evaluation Description
| Evaluation Purpose | To describe CS participation and the development of psychosocial skills and well-being, school engagement, and academic achievement of CS participants. |
| Evaluation Design | Quasi-Experimental: Data were collected on all CS participants enrolled in Boston Public Schools (BPS) in 2003–2004 (Year 3, N = 855). To assess whether CS serves youth who are among the most disadvantaged of all BPS students, participants’ demographic and educational characteristics were compared to the general BPS population in grades 6¬–8 (N = 14,183). In addition, 8th Grade Academy participants’ enrollment in high-quality high schools and on-time promotion to grade 10 were assessed. Surveys were collected from all eligible CS participants attending CS in the spring. To estimate program impacts on school performance, data were collected on a comparison group of youth not participating in CS (n = 855). The core matching criteria included gender, race, grade in school, free or reduced-price lunch (FRPL) eligibility, and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended, and bilingual/special education status. With the exception of similar grade 4 MCAS scores, these data were based on youth’s 1st year in CS. MCAS scores were collected for participants, control group members, and BPS students overall. Youth with high and low CS exposure levels were compared to matched nonparticipants. High exposure was defined as attending 60% or more of the days CS was offered in both the fall and spring. For 8th Grade Academy participants, this requirement further stipulated that this participation rate be maintained for 2 or more years. Low exposure was defined as not meeting these criteria. Demographic characteristics did not significantly differ by exposure level. Exposure level differences are presented only when they are statistically significant. |
| Data Collection Methods | Secondary Source/Data Review: BPS provided data on student achievement (MCAS scores and grades), demographics, and school performance (e.g., attendance, high school choices) for CS participants and BPS students overall. Evaluators continued to access BPS data on youth who stopped attending or graduated from CS as long as they remained in the BPS system. In addition, CS provided program enrollment and attendance data. For the measure of high school choices, BPS high schools were classified as high, medium, or low quality. CS encouraged 8th Grade Academy graduates to attend high- or medium-quality schools. In classifying schools, CS considered each school’s size, MCAS scores, percent of graduates attending college or postsecondary training, attendance and dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). Surveys/Questionnaires: Surveys measured participants’ perceptions of their CS experiences as well as psychosocial well-being and development. Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems. The following scales are embedded in the youth survey: (a) Connection to CS, 5 items measuring agreement with statements such as “I feel like I belong here”; (b) Positive Relationships With Staff, 9 items measuring agreement with statements such as “CS staff always keep their promises”; (c) Peer Support, 3 items measuring agreement with statements such as “Students go out of their way to help each other”; (d) Leadership Opportunities, 4 items measuring frequency of engagement in activities such as “helped plan a CS event or activity”; and (e) Active Learning Opportunities, 18 items measuring the frequency that CS accomplishes things like “really get you thinking.” For all scales, higher numbers indicate greater presence or strength of that characteristic. |
| Data Collection Timeframe | Data were collected during the 2003–2004 year (Year 3). |
Findings:
Formative/Process Findings
| Program Context/ Infrastructure | Average participant survey assessment scores (on a scale of 1–4 , unless otherwise noted) were as follows: (a) Connection to CS: 2.94 for sixth graders, 3.01 for seventh graders, 2.84 for eighth graders; (b) Positive Relationships With Staff: 3.11 for sixth graders, 3.13 for seventh graders, 2.95 for eighth graders; (c) Peer Support: 2.80 for sixth graders, 2.77 for seventh graders, 2.64 for eighth graders; (d) Active Learning Opportunities: 3.00 for sixth graders, 2.99 for seventh graders, 2.81 for eighth graders; and (e) Leadership Opportunities (scale of 1–3): 1.97 for sixth graders, 2.05 for seventh graders, 1.83 for eighth graders. Eighth graders scored approximately 4% lower on each measure than other grades. |
| Recruitment/ Participation |
The CS sample had a significantly higher proportion of African American youth than BPS overall (67% vs. 46%, p < .001). With the exception of Native American students, who made up less than 1% of both populations, BPS overall had significantly higher proportions of students of other races than CS (p < .001), including Hispanic (30% vs. 24%), White (14% vs. 6%), and Asian (9% vs. 3%). Both BPS overall and CS participants were evenly split between girls and boys. Of all CS youth, 30% were eligible to participate in all 3 years that CS was offered, of which 28% participated for all 3 years, 40% in grade 6 only, 20% in grades 6 and 8 only, and 12% in grades 6 and 7 only. |
Summative/Outcome Findings
| Academic | 8th Grade Academy youth attended school significantly more often than matched nonparticipants (p < .01), both overall (92% vs. 88%), and among the CS high-exposure group (95% vs. 90%), although the two groups did not differ on grade 8 school suspension rates, overall or by exposure level. Former 8th Grade Academy participants with high exposure had significantly higher (p < .05) school attendance (94% vs. 88%) and lower suspension (2% vs. 10%) rates in grade 9 than matched nonparticipants; these differences did not hold for CS participants overall or by low exposure. 8th Grade Academy youth were promoted to grade 9 at a significantly higher rate (p < . 01) than matched nonparticipants overall (98% vs. 94%) and among high-exposure groups (100% vs. 92%). Former 8th Grade Academy youth and match nonparticipants did not differ on grade 10 promotion rates. 8th Grade Academy youth had slightly but significantly higher scaled grade 8 MCAS math scores than matched nonparticipants (p < .01 overall, p < .05 among high and low exposure groups). Like comparisons, however, they performed at low levels; over half scored at the Warning level. 8th Grade Academy youth did not differ from matched nonparticipants on grade 8 English and math grades, as measured in the third marking period. Former 8th Grade Academy youth had significantly higher grade 9 English and math grades than matched nonparticipants (p < .05); high-exposure youth also had significantly higher English grades (p < .05), and low-exposure youth had significantly higher math grades (p < .01) than nonparticipants. Former 8th Grade Academy youth selected high quality high schools at a significantly higher rate (p < .001) than matched nonparticipants overall (72% vs. 32%) and among high- and low-exposure (70% vs. 34% and 74% vs. 31%, respectively) groups. First year CS sixth and seventh graders attended school at a significantly higher rate (p < .001) than matched nonparticipants overall (93% vs. 90%) and among the high-exposure group (95% vs. 90%), with an average of 9 more days of school for participants overall. High-exposure seventh graders in their second year of CS had significantly higher school attendance rates than matched nonparticipants (95% vs. 93%, p < .05); this difference did not hold for the sample overall. First year CS sixth and seventh graders were suspended significantly less often (p < .001) than matched nonparticipants overall (10% vs. 16%) and among the high-exposure group (5% vs. 16%). Suspension rates of second year CS seventh graders did not differ from comparisons. First year CS sixth and seventh graders were promoted to the next grade at a significantly higher rate than matched nonparticipants overall (96% vs. 94%, p < .05) and among the high-exposure group (98% vs. 94%, p < .01). Promotion rates of second year CS seventh graders did not differ from comparisons. First year CS sixth and seventh graders overall did not differ from matched nonparticipants in third marking period English and math grades, though high-exposure youth significantly outperformed nonparticipants (p < .001 for English; p < .05 for math). Seventh graders in their second year of CS significantly outperformed matched nonparticipants in math grades overall (p < .01) and among the high-exposure group (p < .05), although they did not differ in English grades. First year CS youth scored significantly higher on the grade 7 MCAS ELA test than matched nonparticipants overall (p < .05) and among the high-exposure group (p < .01), although the two groups did not differ in the percent at the warning level. First year CS youth performed similarly to matched nonparticipants on grade 6 MCAS math scaled scores and the percent scoring at warning level. Youth in their second year of CS were less likely than matched nonparticipants to score at the warning level on the grade 7 MCAS ELA test (p < .01 overall, and p < .05 for high- and low-exposure), although no differences were found in scaled scores. |