The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.
A Profile of the Evaluation of Citizen Schools
Profile Information
- Full Text (HTML)
- Request reprint permission
- How to cite
- About the OST Database
Profile Updated
May 9, 2011
Program Description
| Overview | Citizen Schools (CS) operates a national network of apprenticeship programs for middle school students that connects adult volunteers to youth in hands-on afterschool learning projects. The CS programs are designed to help youth develop academic and leadership skills needed to succeed in school, get into college, and become leaders in their careers and in their communities. |
| Start Date | 1995 |
| Scope | national |
| Type | afterschool |
| Location | urban |
| Setting | public school |
| Participants | middle school students (grades 6–8) |
| Number of Sites/Grantees | 30 sites in 2006–2007 nationwide; 37 sites nationwide (2009–2010) |
| Number Served | approximately 3,000 in 2006–2007 nationwide; approximately 4,000 nationwide (2009–2010) |
| Components | Twice a week after school, CS youth participate in apprenticeships, which consist of hands-on learning projects led by volunteer Citizen Teachers. Apprentices work collaboratively in small groups to build solar cars, litigate mock trials, publish children's books, and so on. On other weekday afternoons, CS staff lead structured afterschool activities to enhance school success of youth, working on homework and building organizational and study skills to help youth navigate their education through high school, graduation, and beyond. Each semester culminates in “WOW!”—a public presentation of the CS participants’ projects. In 2001–2002, CS launched its 8th Grade Academy program in Boston for eighth graders who began CS in a prior year. In addition to offering activities similar to those at other CS campuses, 8th Grade Academy is intended to help youth apply to and succeed in competitive high schools and to introduce them to the college application process through experiential learning activities that build academic and life skills and give youth access to coaches, technology, internships, and other educational programs. Each participant is assigned a writing coach (typically a local lawyer). In 2004, CS launched an alumni program to support Academy graduates during the high school transition period. |
| Funding Level | approximately $20.9 million in 2009–2010 |
| Funding Sources | Major funders include ArcLight Capital Partners, The Atlantic Philanthropies, Bank of America, Josh & Anita Bekenstein, Edna McConnell Clark Foundation, The John S. and James L. Knight Foundation, The Koogle Foundation, The Lovett-Woodsum Foundation, The Picower Foundation, The Samberg Family Foundation, Skoll Foundation, AmeriCorps, State of North Carolina Department of Social Servies, Boston Public Schools, Duke Endowment, Houston Independent School District, Nellie Mae Education Foundation, and many other individual, corporate, and philanthropic donors. |
Evaluation
| Overview | A 7-year evaluation (2001–2008) focused on the program experiences and outcomes of 5 cohorts of participants at the Boston site. |
| Evaluators | Policy Studies Associates, Inc. |
| Evaluations Profiled | Evidence from Two Student Cohorts on the Use of Community Resources to Promote Youth Development: Phase II Report Putting Students on a Pathway to Academic and Social Success: Phase III Findings Preparing Students in the Middle Grades to Succeed in High School: Findings from Phase IV Progress Toward High School Graduation: Youth Outcomes in Boston Achieving High School Graduation: Citizen Schools’ Youth Outcomes in Boston |
| Evaluations Planned | Citizen Schools is proceeding with a new evaluation plan in which Abt Associates Inc. and Public/Private Ventures will serve as external evaluators. |
| Report Availability | Fabiano, L., Espino, J., & Reisner, E. R., with Pearson, L. M. (2003). Citizen Schools: Using community resources to promote youth development. Phase I Report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2003.pdf Espino, J., Fabiano, L., & Pearson, L. M. (with Kirkwood K. P., Afolabi, K., & Pasatta, K.). (2004). Citizen Schools: Evidence from two student cohorts on the use of community resources to promote youth development. Phase II report of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Fabiano, L., Pearson, L. M., & Williams, I. J. (2005). Putting students on a pathway to academic and social success: Phase III findings of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2005.pdf Fabiano, L., Pearson, L. M., Reisner, E. R., & Williams, I. J. (2006). Preparing students in the middle grades to succeed in high school: Findings from Phase IV of the Citizen Schools evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2006.pdf Pearson, L. M., Vile, J. D., & Reisner, E. R. (2008). Establishing a foundation for progress toward high school graduation: Findings from Phase V of the Citizen Schools Evaluation. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2008.pdf Vile, J. D., Arcaira, E., & Reisner, E. R. (2009). Progress toward high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates. Available at: www.emcf.org/fileadmin/user/PDF/Results/eval_CitizenSchoolsEvaluation2009.pdf Arcaira, E., Vile, J. D., & Reisner, E. R. (2010). Achieving high school graduation: Citizen Schools’ youth outcomes in Boston. Washington, DC: Policy Studies Associates. Available at: www.policystudies.com/studies/?download=79&id=39 |
Contacts
| Evaluation | Erikson Arcaira Research Associate Policy Studies Associates, Inc. 1718 Connecticut Avenue, NW, Suite 400 Washington, DC 20009 Tel: 202-939-5343 Fax: 202-939-5732 Email: earcaira@policystudies.com |
|
| Program | Michael Kubiak Director of Research and Evaluation Citizen Schools 308 Congress Street, 5th Floor Boston, MA 02210 Tel: 617-695-2300 ext. 132 Fax: 617-695-2367 Email: michaelkubiak@citizenschools.org |
|
| Profile Updated | May 9, 2011 | |
Evaluation 1: Evidence From Two Student Cohorts on the Use of Community Resources to Promote Youth Development. Phase II Report
Evaluation Description
| Evaluation Purpose | To determine whether program services were implemented as intended and, if so, whether the program had a positive impact on participants’ academic achievement, school engagement, and psychosocial skills and well-being. |
| Evaluation Design | Quasi-Experimental and Non-Experimental: Data were collected on 2 cohorts of youth in 8 Boston CS campuses. Cohort I consisted of a sample of 253 sixth and eighth graders attending CS in Fall 2001 (Year 1). Cohort II consisted of a sample of 323 sixth and eighth graders attending CS in Fall 2002 (Year 2). To assess whether CS serves youth who were among the most disadvantaged of all BPS students, the demographic and educational characteristics of participants were compared to the general Boston Public Schools (BPS) student population. Baseline and follow-up surveys were collected from all eligible CS participants attending CS at the time of each survey. Survey data from program staff (campus directors, deputy directors, and CS teaching associates), volunteers (Citizen Teachers), and parents were collected for Years 1 and 2. To estimate program impacts on school-related performance, data were collected on a comparison group of 225 matched nonparticipants for Cohort I, and 260 matched nonparticipants for Cohort II. The core matching criteria included gender, race, grade in school, eligibility for free or reduced-price lunch (FRPL), and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended, bilingual education status, and special education status. With the exception of constants like grade 4 MCAS scores, these data were based on youth’s 1st year in CS. MCAS scores were collected for both participants and control group members. Matched nonparticipants were identical to participants on the core matching criteria. Eighty-three percent of Cohort I sixth graders and 81% of Cohort I eighth graders as well as 90% of Cohort II sixth graders and 78% of Cohort II eighth graders scored within 8 points of their matched nonparticipants on the MCAS. Twenty-six percent of matched nonparticipants attended the same school as their participants. The majority of matched nonparticipants were similar to their participants with respect to their bilingual education and special education status. Evaluators also compared the following subgroups of participants to their nonparticipants counterparts: those with high CS exposure (attending over 80% of CS days), grade level, cohort, and those in CS programs offering special activities (e.g., special writing courses). |
| Data Collection Methods | Observation: Evaluators visited each of the 8 campuses for 2 days in spring 2002. Secondary Source/Data Review: Data were collected from BPS records for Year 1 CS participants (60 Cohort I and 260 Cohort II youth) and BPS students overall on the following: demographics, school performance (e.g., attendance, high school choices), and academic achievement (i.e., grade 4 MCAS test scores and first marking period English and math grades). Evaluators continued to access BPS data on youth who stopped attending or graduated from CS as long as they remained in the BPS system. For the measure of high school choices, BPS high schools were classified as high, medium, or low quality. CS encouraged 8th Grade Academy graduates to attend high- or medium-quality schools. In classifying schools, CS considered each school’s size, MCAS performance, percent of graduates attending college or postsecondary training, attendance and dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). CS provided program enrollment and attendance data. Surveys/Questionnaires: articipant surveys examined youth’s CS experiences and their psychosocial well-being and development. Cohort I baseline surveys were completed by 234 youth in Fall 2001, of which 145 completed the first follow-up in spring 2002, and 31 completed a second follow-up in Spring 2003. Cohort II baseline surveys were completed by 323 youth in Fall 2002, of which 187 completed a follow-up in Spring 2003. Staff surveys asked about roles and responsibilities in CS, training activities, parent involvement, staff meetings, program planning and activities, use of student teams, interactions with participants’ schools, and teaching methods. Parent surveys asked about demographics and family background, their children’s participation in CS, perceptions of and interactions with CS, and perceptions of program impacts on their children. Volunteer surveys asked about apprenticeships offered, training and support from CS, planning and preparation, teaching methods, perceptions of volunteers’ overall experiences, and background questions. Staff and parent surveys were administered once a year in the spring. The volunteer survey was administered in Fall 2001, Spring 2002, and Spring 2003. This report mainly uses Year 2 survey data, but provides some comparisons with Year 1 survey data. In Year 2, 228 parents, 57 staff members (8 campus directors and deputy directors and 49 teaching associates), and 43 volunteers completed surveys. Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems. The following scales are embedded in youth surveys: (a) Connection to CS, 5 items measuring agreement with statements such as “I feel like I belong here”; (b) Positive Relationships With Staff, 9 items measuring agreement with statements such as “CS staff always keep their promises”; (c) Peer Support, 3 items measuring agreement with such statements as “Students go out of their way to help each other”; (d) Leadership Opportunities, 4 items measuring frequency of engaging in activities such as “helped plan a CS event or activity like an Exploration or a Potluck Dinner”; (e) Active Learning Opportunities, 18 items measuring the frequency that CS activities, staff, and volunteers accomplish things like “make learning fun for you”; (f) Self-Efficacy, 8 items measuring agreement with statements such as “If something looks too hard, I will not even bother to try it” and 9 items measuring how sure youth are that things will “work out well” when faced with occurrences such as having “to learn something new at school”; (g) Locus of Control, 4 items measuring agreement with statements such as “In my life, good luck is more important than hard work for success”; (h) Social Competence, 5 items measuring agreement with statements such as “I can make suggestions to other students without being bossy”; (i) Positive Interpersonal Behavior, 6 items measuring frequency of engaging in behaviors such as trying to understand why others feel a certain way when they have problems with other youth; (j) Relationship With Parents or Guardians and Relationship With Adults, two 4-item scales measuring frequency of talking with parents/guardians or other adults about topics such as what is going on in their lives; and (k) Future Optimism, 10 items measuring chances of accomplishing things such as graduating from high school. For all scales, a rating of 1 indicates the absence or lowest level of that characteristic, with higher numbers indicating greater presence or strength of that characteristic. |
| Data Collection Timeframe | Data were collected during the 2001–2002 (Year 1) and 2002–2003 (Year 2) years; data refer to Year 2 unless otherwise indicated. |
Findings:
Formative/Process Findings
| Activity Implementation | Depending on the campus, program activities were offered from 2.25 hours to 4 hours per day, with an average of 3.5 to 4 hours per day, over an 8- to 10-month period. On average, programs offered activities on 81 days, with six campuses offering 3 days of activities each week, and two offering 4 days. Activities were mainly offered weekdays after school, but two campuses also offered activities on Saturdays. All campuses held apprenticeships once a week for 1.5 to 2.75 hours, in three to eight areas each semester, mostly in the fields of arts, law, and business. Writing and data analysis/math projects were each offered once a week for 1.25 hours to just under 3.5 hours; six campuses offered these projects at least 1.5 hours a week. All campuses offered explorations of local institutions, neighborhoods, and universities once a week for anywhere from just under an hour to approximately 3 hours, with half of the campuses offering fewer than 1.5 hours and the other half offering more than 2 hours. Five campuses offered homework time every day of programming, while the remaining 3 offered this activity on all but one program day. The weekly amount of homework time ranged from just under 2 hours to nearly 5 hours. Compared to other CS activities, campus directors rated their implementation of explorations and apprenticeships as more successful and reported more challenges with writing and data analysis/math projects and homework. Directors cited youth absenteeism and lack of time as the most pervasive challenges in implementing program elements. |
| Parent/Community Involvement | The majority of surveyed parents reported that staff contacted them at least twice a month to discuss their child’s progress in the program (73%) and academic performance (62%). Parents also reported that staff requested their attendance or assistance at events such as parent meetings or potluck dinners. Analyses of parent survey data and CS characteristics revealed that staff’s prior experience working with youth was positively and significantly related to the frequency with which staff talked to parents about their child’s academic progress (p = 0.03). As the number of hours of exploration activities increased, parents’ reports of the frequency with which staff talked to them about their child’s progress in CS also significantly increased (p = 0.02). The activities in which the greatest numbers of parents reported being involved were campus registration/orientation meetings (54%) and potluck dinners (53%). The activity that parents reported participating in least was helping to organize a family event (3%). Campuses that provided the most hours of academic skill-building tended to have significantly more parents reporting attending a CS orientation/registration meeting (p = 0.03) or a potluck dinner (p = 0.01). Also, campuses that employed staff with more prior CS experience tended to have significantly more parents reporting attending a CS potluck dinner than campuses with less experienced staff (p = 0.00). |
| Program Context/ Infrastructure | During their 1st year of CS, Cohort I and II participants scored an average of 2.74 out of 4 on the Peer Support scale on 1st year follow-up surveys. Among Cohort I sixth graders who participated for a 2nd year, the average score on this scale was 2.68. First year follow-up surveys indicated that during their 1st year of CS, Cohort I and II participants scored an average of 1.90 out of 3 on the Leadership Opportunities scale. The average score of Cohort I sixth graders who participated for a 2nd year was 2.04. A positive relationship was found between scores on the Leadership Opportunities index and the number of hours of exploration activities offered and staff with prior experience in CS. In contrast, a negative relationship was found between scores on this measure and the number of youth per CS staff member and hours of academic skill-building activities. The average score for Cohort I and II participants during their 1st year of participation on the Active Learning Opportunities scale was 2.89 out of 4. The average score was 2.98 among Cohort I sixth graders who participated for a 2nd year. |
| Recruitment/ Participation | CS participants, like BPS students overall, were about evenly split between males and females. African Americans made up 69% of participants, a higher percentage than for BPS students overall (47%). Hispanics were the next largest ethnic group among participants (23%) compared to 29% of BPS students overall. Whites, Asians, and Native Americans together constituted 9% of all participants. In contrast, 14% of BPS students were White, 9% were Asian, and less than 1% were Native American. A higher percentage of CS participants than BPS students overall were FRPL-eligible (90% vs. 73%) and in special education programs (23% vs. 19%). However, a smaller percentage of CS participants were enrolled in a bilingual education program than BPS students overall (5% vs. 10%). Of surveyed parents, 21% reported something other than a job as their primary source of income and 32% reported completing college (10% of which also had a graduate degree). Both CS participants and BPS students overall attended school 93–95% of the time. Fourteen percent of CS participants were suspended at some point during the year. Less than half (45%) of Cohort II CS sixth graders performed at the standard of needs improvement or better on the MCAS math test as fourth graders, with only 5% performing at a proficient or advanced level; the remaining 55% performed at the warning level. In contrast, 42% of BPS sixth graders performed at the warning level on the 2001 MCAS math test as fourth graders. Among Cohort II CS eighth graders, 61% performed at the needs improvement level or better, with 14% performing at the proficient level. Thirty-nine percent of CS eighth graders performed at the warning level, compared to 43% of BPS eighth graders who took the test as fourth graders in 1999. Among Cohort II CS participants with available data, the average first marking period math grade was a C- for sixth graders (15% failed the period, and only 34% earned a B- or better) and a C for eighth graders (15% failed the period, and 38% passed with a B- or better). About a quarter of Cohort II CS participants performed at the warning level on the ELA MCAS test (28% of sixth graders and 25% of eighth graders), with similar proportions at the warning level for BPS overall. However, a higher percentage of BPS sixth graders (24%) performed at proficient or advanced levels than CS sixth graders (13%). Among Cohort II CS participants with available data, both sixth and eighth graders had an average first marking period English grade of C; 7% of sixth graders and 10% of eighth graders failed the marking period while 45% of sixth graders and 34% of eighth graders had a B- or higher. The number of days a youth attended CS in Year 2 ranged from 1 to 98 (median = 62). Two thirds of youth participated for at least 60% of the days the program was offered, while 50% of youth participated for 80% or more days. In Year 2, 68% of youth enrolled in CS were enrolled for the duration of the fall and continued to be enrolled at the time of survey administration in the spring. Nearly equal percentages of youth withdrew from CS during the fall semester, between the Fall 2002 and Spring 2003 semesters, and during the Spring 2003 semester (8–10%). Youth who continued to participate in CS for a 2nd year had significantly higher baseline Locus of Control scale scores than those who withdrew (2.91 vs. 2.72 out of 4, p = .03). When asked on the survey how often they missed a CS activity due to arriving late or leaving early, 48% of participants reported never, 22% reported usually or almost always, while the remaining youth reported sometimes. |
| Satisfaction | The average score on a survey item measuring youth satisfaction with CS in Year 1 was 3.22 out of 5. The average score for Cohort I youth who returned for a 2nd year was 3.44. The average youth score on the Connection to CS scale was 2.87 out of 4 in Year 1. The average for Cohort I sixth graders who continued to participate for a 2nd year was 3.04. The average parent survey score of overall satisfaction with CS was 4.04 out of 5. Parents of children at campuses that offered more hours of academic skill-building were more satisfied than parents of children a campuses with fewer hours of such activities, though this difference was only marginally significant (p < .10). Parents of children at campuses that offered more hours of interpersonal skill-building were significantly less satisfied than parents of children at campuses with fewer hours of such activities (p = 0.03). |
| Staffing/Training | The youth-to-staff ratio averaged 10:1 across campuses. The majority of CS staff had a college degree or above—all leadership staff (campus directors and deputy directors) and 86% of Citizen Teachers had at least a college degree; 51% of Citizen Teachers also had a graduate degree. The majority of CS staff were also female (especially leadership staff). Direct service staff tended to be under 25 years old and White; leadership staff were more likely than other staff to be older than 25 and non-White. In Year 2, almost two thirds of staff had experience working with CS, an increase of 24 percentage points from Year 1. The average survey rating of staff/volunteer enjoyment of working at CS was 3.61 out of 4. The average survey rating of staff training and support satisfaction was 2.75 out of 4. The average score on the Relationships With Staff scale was 3.02 out of 4 for Cohort I and II youth during their 1st year of CS participation. Among Cohort I CS sixth graders who continued to participate for another year, the average score on this scale was 3.15. |
Summative/Outcome Findings
| Academic | A significantly greater percentage of 8th Grade Academy graduates selected high-quality high schools than matched nonparticipants (70% vs. 12% for Cohort I, and 70% vs. 46% for Cohort II, p = .001). A higher percentage of Cohort I 8th Grade Academy participants were promoted to grade 10 on time (i.e., within 2 years) than matched nonparticipants (87% vs. 74%); this difference was marginally significant (p = 0.07). A significantly higher percentage of high exposure participants were also promoted to grade 10 on time than matched nonparticipants (90% vs. 73%, p = 0.05). In Year 1, CS sixth graders overall performed significantly better than matched nonparticipants on attendance, suspension rates, and grade promotion (p < .05). Sixth graders with high CS exposure outperformed matched nonparticipants on attendance, suspension rates, math and English grades, and promotion (p < .05). Seventh graders in Cohort I outperformed matched nonparticipants on suspension rates, MCAS ELA scores, and math course grades. CS eighth graders performed better than matched nonparticipants on attendance. In their ninth grade year, Cohort I 8th Grade Academy graduates did not significantly outperform matched nonparticipants on any of the evaluation’s indicators. |
| Youth Development | Cohort I youth's scores on the Locus of Control scale, and a “confidence in public speaking” sub item of the Self-Efficacy scale, demonstrated a marginally significant increase from Fall 2001 to Spring 2003 (p < .10). They did not exhibit any changes on the two overall Self-Efficacy scales, or on a “confidence in writing” sub item. Cohort II youth did not increase any of their self-efficacy/confidence scores from baseline to first follow-up. No significant changes were seen from Fall 2002 to Spring 2003 for either CS cohort on scales of Social Competency, Positive Interpersonal Behavior, Relationship with Parents or Guardians, Future Optimism, or importance of school for future success sub items. Comparisons of the average difference between baseline and first-year follow-up scores for Cohort I and II revealed that in Years 1 and 2, participants experienced similar changes on measures of psychosocial skills and well-being. |
Evaluation 2: Putting Students on a Pathway to Academic and Social Success: Phase III Findings of the Citizen Schools Evaluation
Evaluation Description
| Evaluation Purpose | To describe CS participation and the development of psychosocial skills and well-being, school engagement, and academic achievement of CS participants. |
| Evaluation Design | Quasi-Experimental: Data were collected on all CS participants enrolled in Boston Public Schools (BPS) in 2003–2004 (Year 3, N = 855). To assess whether CS serves youth who are among the most disadvantaged of all BPS students, participants’ demographic and educational characteristics were compared to the general BPS population in grades 6¬–8 (N = 14,183). In addition, 8th Grade Academy participants’ enrollment in high-quality high schools and on-time promotion to grade 10 were assessed. Surveys were collected from all eligible CS participants attending CS in the spring. To estimate program impacts on school performance, data were collected on a comparison group of youth not participating in CS (n = 855). The core matching criteria included gender, race, grade in school, free or reduced-price lunch (FRPL) eligibility, and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended, and bilingual/special education status. With the exception of similar grade 4 MCAS scores, these data were based on youth’s 1st year in CS. MCAS scores were collected for participants, control group members, and BPS students overall. Youth with high and low CS exposure levels were compared to matched nonparticipants. High exposure was defined as attending 60% or more of the days CS was offered in both the fall and spring. For 8th Grade Academy participants, this requirement further stipulated that this participation rate be maintained for 2 or more years. Low exposure was defined as not meeting these criteria. Demographic characteristics did not significantly differ by exposure level. Exposure level differences are presented only when they are statistically significant. |
| Data Collection Methods | Secondary Source/Data Review: BPS provided data on student achievement (MCAS scores and grades), demographics, and school performance (e.g., attendance, high school choices) for CS participants and BPS students overall. Evaluators continued to access BPS data on youth who stopped attending or graduated from CS as long as they remained in the BPS system. In addition, CS provided program enrollment and attendance data. For the measure of high school choices, BPS high schools were classified as high, medium, or low quality. CS encouraged 8th Grade Academy graduates to attend high- or medium-quality schools. In classifying schools, CS considered each school’s size, MCAS scores, percent of graduates attending college or postsecondary training, attendance and dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). Surveys/Questionnaires: Surveys measured participants’ perceptions of their CS experiences as well as psychosocial well-being and development. Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems. The following scales are embedded in the youth survey: (a) Connection to CS, 5 items measuring agreement with statements such as “I feel like I belong here”; (b) Positive Relationships With Staff, 9 items measuring agreement with statements such as “CS staff always keep their promises”; (c) Peer Support, 3 items measuring agreement with statements such as “Students go out of their way to help each other”; (d) Leadership Opportunities, 4 items measuring frequency of engagement in activities such as “helped plan a CS event or activity”; and (e) Active Learning Opportunities, 18 items measuring the frequency that CS accomplishes things like “really get you thinking.” For all scales, higher numbers indicate greater presence or strength of that characteristic. |
| Data Collection Timeframe | Data were collected during the 2003–2004 year (Year 3). |
Findings:
Formative/Process Findings
| Program Context/ Infrastructure | Average participant survey assessment scores (on a scale of 1–4 , unless otherwise noted) were as follows: (a) Connection to CS: 2.94 for sixth graders, 3.01 for seventh graders, 2.84 for eighth graders; (b) Positive Relationships With Staff: 3.11 for sixth graders, 3.13 for seventh graders, 2.95 for eighth graders; (c) Peer Support: 2.80 for sixth graders, 2.77 for seventh graders, 2.64 for eighth graders; (d) Active Learning Opportunities: 3.00 for sixth graders, 2.99 for seventh graders, 2.81 for eighth graders; and (e) Leadership Opportunities (scale of 1–3): 1.97 for sixth graders, 2.05 for seventh graders, 1.83 for eighth graders. Eighth graders scored approximately 4% lower on each measure than other grades. |
| Recruitment/ Participation |
The CS sample had a significantly higher proportion of African American youth than BPS overall (67% vs. 46%, p < .001). With the exception of Native American students, who made up less than 1% of both populations, BPS overall had significantly higher proportions of students of other races than CS (p < .001), including Hispanic (30% vs. 24%), White (14% vs. 6%), and Asian (9% vs. 3%). Both BPS overall and CS participants were evenly split between girls and boys. Of all CS youth, 30% were eligible to participate in all 3 years that CS was offered, of which 28% participated for all 3 years, 40% in grade 6 only, 20% in grades 6 and 8 only, and 12% in grades 6 and 7 only. |
Summative/Outcome Findings
| Academic | 8th Grade Academy youth attended school significantly more often than matched nonparticipants (p < .01), both overall (92% vs. 88%), and among the CS high-exposure group (95% vs. 90%), although the two groups did not differ on grade 8 school suspension rates, overall or by exposure level. Former 8th Grade Academy participants with high exposure had significantly higher (p < .05) school attendance (94% vs. 88%) and lower suspension (2% vs. 10%) rates in grade 9 than matched nonparticipants; these differences did not hold for CS participants overall or by low exposure. 8th Grade Academy youth were promoted to grade 9 at a significantly higher rate (p < . 01) than matched nonparticipants overall (98% vs. 94%) and among high-exposure groups (100% vs. 92%). Former 8th Grade Academy youth and match nonparticipants did not differ on grade 10 promotion rates. 8th Grade Academy youth had slightly but significantly higher scaled grade 8 MCAS math scores than matched nonparticipants (p < .01 overall, p < .05 among high and low exposure groups). Like comparisons, however, they performed at low levels; over half scored at the Warning level. 8th Grade Academy youth did not differ from matched nonparticipants on grade 8 English and math grades, as measured in the third marking period. Former 8th Grade Academy youth had significantly higher grade 9 English and math grades than matched nonparticipants (p < .05); high-exposure youth also had significantly higher English grades (p < .05), and low-exposure youth had significantly higher math grades (p < .01) than nonparticipants. Former 8th Grade Academy youth selected high quality high schools at a significantly higher rate (p < .001) than matched nonparticipants overall (72% vs. 32%) and among high- and low-exposure (70% vs. 34% and 74% vs. 31%, respectively) groups. First year CS sixth and seventh graders attended school at a significantly higher rate (p < .001) than matched nonparticipants overall (93% vs. 90%) and among the high-exposure group (95% vs. 90%), with an average of 9 more days of school for participants overall. High-exposure seventh graders in their second year of CS had significantly higher school attendance rates than matched nonparticipants (95% vs. 93%, p < .05); this difference did not hold for the sample overall. First year CS sixth and seventh graders were suspended significantly less often (p < .001) than matched nonparticipants overall (10% vs. 16%) and among the high-exposure group (5% vs. 16%). Suspension rates of second year CS seventh graders did not differ from comparisons. First year CS sixth and seventh graders were promoted to the next grade at a significantly higher rate than matched nonparticipants overall (96% vs. 94%, p < .05) and among the high-exposure group (98% vs. 94%, p < .01). Promotion rates of second year CS seventh graders did not differ from comparisons. First year CS sixth and seventh graders overall did not differ from matched nonparticipants in third marking period English and math grades, though high-exposure youth significantly outperformed nonparticipants (p < .001 for English; p < .05 for math). Seventh graders in their second year of CS significantly outperformed matched nonparticipants in math grades overall (p < .01) and among the high-exposure group (p < .05), although they did not differ in English grades. First year CS youth scored significantly higher on the grade 7 MCAS ELA test than matched nonparticipants overall (p < .05) and among the high-exposure group (p < .01), although the two groups did not differ in the percent at the warning level. First year CS youth performed similarly to matched nonparticipants on grade 6 MCAS math scaled scores and the percent scoring at warning level. Youth in their second year of CS were less likely than matched nonparticipants to score at the warning level on the grade 7 MCAS ELA test (p < .01 overall, and p < .05 for high- and low-exposure), although no differences were found in scaled scores. |
Evaluation 3: Preparing Students in the Middle Grades to Succeed in High School: Findings from Phase IV
Evaluation Description
| Evaluation Purpose | To analyze program impacts on former participants’ school engagement and academic achievement in 9th and 10th grades over time. |
| Evaluation Design | Quasi-Experimental: Data were collected on 8th Grade Academy youth who attended Boston Public Schools (BPS) in one of four cohorts: Year 1 (2001–2002) through Year 4 (2004–2005). Each participant was matched to 3 nonparticipants who attended BPS, based on five core matching variables: gender, race, grade in school, free or reduced-price lunch (FRPL) eligibility, and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended, and bilingual/special education status. With the exception of constants like grade 4 MCAS scores, these data were based on the year prior to each youth’s participation in the evaluation. Each participant’s 3 matches were ranked based on how similar they were to the participant on the matching criteria. Participants were compared to their most similar nonparticipant. Matched nonparticipants and participants were similar on all matching criteria. A participant’s matched nonparticipant sometimes changed from year to year; the second or third most similar match sometimes had to be used when a matched nonparticipant left the BPS system or was not promoted on schedule. Between Years 1 and 4, 354 youth participated in 8th Grade Academy and had parental consent to be included in the evaluation; 48 participated in Year 1, 85 in Year 2, 118 in Year 3, and 103 in Year 4. To be included in the analyses, youth had to be promoted to grade 9 at the end of the year in which they participated in 8th Grade Academy; 97% of the sample did so. Grade 9 promotion did not vary significantly by class year nor between participants and nonparticipants. Analyses focused on former 8th Grade Academy youth as they transitioned to high school and progressed through grades 9 and 10. Youth with high and low CS-exposure levels were compared to their matched nonparticipants. High exposure was defined as 2 or more years of CS participation at an attendance rate of 60% or more each semester for all semesters. Low exposure was defined as not meeting these criteria. Of former 8th Grade Academy participants, 49% were classified as high exposure. Exposure level differences are presented only when they are statistically significant. |
| Data Collection Methods | Secondary Source/Data Review: BPS and four Boston-area charter schools provided data on student achievement (MCAS scores and course grades), demographics, and school performance (e.g., attendance, high school choices). For the measure of high school choices, BPS high schools were classified as high, medium, or low quality. CS encouraged 8th Grade Academy graduates to attend high- or medium-quality schools. In classifying schools, CS considered school size, MCAS scores, percent of graduates attending college or postsecondary training, attendance, dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems. |
| Data Collection Timeframe | Data covered the 2001–2002 academic year (Year 1) through the 2004–2005 academic year (Year 4). |
Findings:
Summative/Outcome Findings
| Academic | Former 8th Grade Academy participants selected high-quality high schools at a significantly higher rate than their matched nonparticipants, overall (65% vs. 26%, p < .001) and among both the high-exposure (65% vs. 22%) and the low-exposure (64% vs. 31%) groups. Former 8th Grade Academy participants attended school at significantly higher rates in their ninth grade year than matched nonparticipants (87% vs. 82%, p < .05), for a difference of approximately 9 days of school. Former 8th Grade Academy youth with high exposure (but not overall) had significantly lower suspension rates in their ninth grade year than matched nonparticipants (1% vs. 10%, p < .01). Former 8th Grade Academy participants had significantly better (p < .05) grade 9 English course grades than matched nonparticipants, both overall and among the high exposure group. They did not significantly differ on average grade 9 math course grades. Former 8th Grade Academy youth with high exposure (but not overall) were promoted on time to grade 10 at a significantly higher rate than matched nonparticipants (92% vs. 81%, p < .05). Former 8th Grade Academy participants had significantly higher grade 10 math course grades than matched nonparticipants (p < .05). The two groups did not differ on other grade 10 measures. |
Evaluation 4: Progress Toward High School: Youth Outcomes in Boston
Evaluation Description
| Evaluation Purpose | To examine academic engagement and achievement, including persistence in a high-quality high school and 4-year graduation rate. |
| Evaluation Design | Quasi-Experimental: Data were collected on 8th Grade Academy youth who attended Boston Public Schools (BPS) in one of five cohorts: Year 1 (2001–2002) through Year 5 (2005–2006). Each participant was matched to 3 nonparticipants who attended BPS, based on five core matching variables: gender, race, grade in school, free or reduced-price lunch (FRPL) eligibility, and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended, and bilingual/special education status. With the exception of constants like grade 4 MCAS scores, these data were based on the year prior to each youth’s participation in the evaluation. Each participant’s 3 matches were ranked based on how similar they were to the participant on the matching criteria. Participants were compared to their most similar nonparticipant. Matched nonparticipants and participants were similar on all matching criteria. A participant’s matched nonparticipant sometimes changed from year to year; the second or third most similar match sometimes had to be used when a matched nonparticipant left the BPS system or was not promoted on schedule. Between Year 1 and Year 5, 448 youth participated in 8th Grade Academy and had parental consent to be included in the evaluation; 48 participated in Year 1, 85 in Year 2, 118 in Year 3, 103 in Year 4, and 94 in Year 5. To be included in the analyses, youth had to be promoted to grade 9 at the end of the year in which they participated in 8th Grade Academy; 97% of the sample did so. Grade 9 promotion did not vary significantly by class year, nor between participants and nonparticipants. Analyses focused on former 8th Grade Academy youth as they transitioned to high school and progressed through grade 12. |
| Data Collection Methods | Secondary Source/Data Review: BPS and four Boston-area charter schools provided data on student achievement (MCAS scores and course grades), demographics, and school performance (e.g., attendance, high school choices). For the measure of high school choices, BPS high schools were classified as high, medium, or low quality. CS encouraged 8th Grade Academy graduates to attend high- or medium-quality schools. In classifying schools, CS considered school size, MCAS scores, percent of graduates attending college or postsecondary training, attendance, dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems. |
| Data Collection Timeframe | Data covered the 2001–2002 academic year (Year 1) through the 2006–2007 academic year (Year 6). |
Findings:
Summative/Outcome Findings
| Academic | Participants selected high-quality high schools at a significantly higher rate than did matched nonparticipants (59% vs. 28%, p < .001). Among students entering a high-quality high school in 9th grade, participants persisted in a high quality high school from 9th to 11th grade at a significantly higher rate than matched nonparticipants (72% vs. 41%, p < .01). Former participants attended school significantly more often than did matched nonparticipants in all high school grade levels. Differences ranged from a low of 3 percentage points—equivalent to about 5 additional school days (p < .05)—in 10th grade to a high of 7 percentage points—equivalent to about 13 additional school days (p < .01)—in 11th grade. No significant differences were found in suspension rates between former participants and nonparticipants. Former participants had significantly higher pass rates in their math courses than did matched nonparticipants at grades 9 (73% vs. 65%, p < .05), 10 (79% vs. 64%, p < .01), 11 (83% vs. 72%, p < .05), and 12 (85% vs. 70%, p < .05). Former participants significantly outperformed nonparticipants in earning As and Bs in math courses in grades 9 (34% vs. 24%, p < .05) and 12 (43% vs. 23%, p < .05). Rates of As and Bs did not significantly differ in grades 10 and 11. Former participants had significantly higher pass rates (93% vs. 86%, p < .05) and proficiency rates (53% vs. 40%, p < .01) than did nonparticipants on the 10th grade mathematics MCAS. Former participants achieved significantly higher pass rates in their ELA courses than did matched nonparticipants in grades 9 (82% vs. 69%, p < .01) and 12 (91% vs. 70%, p < .01). Pass rates did not significantly differ in grades 10 and 11. Former participants earned As and Bs in English at significantly higher rates than their matches in grades 10 (45% vs. 29%, p < .01) and 12 (51% vs. 32%, p < .01). Rates of As and Bs did not significantly differ in grades 9 and 11. Former participants passed the ELA MCAS at rates similar to matched nonparticipants but significantly outperformed their matches in attaining proficient and advanced performance levels (48% vs. 39%, p < .05). Overall, former participants were promoted from 9th grade to 10th grade on time at a significantly higher rate than were matched nonparticipants (83% vs. 76%, p < .05). Former participants were promoted on time to 11th and 12th grade at rates similar to their matches. Being “on track” to graduate was defined as being promoted to 10th grade on time and not failing a core course in math or ELA; a significantly larger proportion of former CS participants were on track than were matched nonparticipants (63% vs. 51%, p < .01). For the graduating class of 2007, former 8th Grade Academy participants graduated in 4 years at a significantly higher rate than did the overall BPS class (75% vs. 58%, p < .01). |
Evaluation 5: Achieving High School Graduation: Citizen Schools’ Youth Outcomes in Boston
Evaluation Description
| Evaluation Purpose | To explore whether former 8th Grade Academy participants achieve positive academic outcomes and successfully complete high school. |
| Evaluation Design | Quasi-Experimental: Data were collected on 8th Grade Academy youth who attended Boston Public Schools (BPS) in one of five cohorts: Year 1 (2001–2002) through Year 5 (2005–2006). Each participant was matched to 3 nonparticipants who attended BPS, based on five core matching variables: gender, race, grade in school, free or reduced-price lunch (FRPL) eligibility, and grade 4 Massachusetts Comprehensive Assessment System (MCAS) test scores in math and English language arts (ELA). When possible, youth were also matched on school attended and bilingual/special education status. With the exception of constants like grade 4 MCAS scores, these data were based on the year prior to each youth’s participation in the evaluation. Each participant’s 3 matches were ranked based on how similar they were to the participant on the matching criteria. Participants were compared to their most similar nonparticipant. Matched nonparticipants and participants were similar on all matching criteria. A participant’s matched nonparticipant sometimes changed from year to year; the second or third most similar match sometimes had to be used when a matched nonparticipant left the BPS system or was not promoted on schedule. Between Year 1 and Year 5, 448 youth participated in 8th Grade Academy and had parental consent to be included in the evaluation; 48 participated in Year 1, 85 in Year 2, 118 in Year 3, 103 in Year 4, and 94 in Year 5. To be included in the analyses, youth had to be promoted to grade 9 at the end of the year in which they participated in 8th Grade Academy; 97% of the sample did so. Grade 9 promotion did not vary significantly by class year, nor between participants and nonparticipants. Analyses focused on former 8th Grade Academy youth as they transitioned to high school and progressed through grade 12. |
| Data Collection Methods |
Secondary Source/Data Review: BPS and four Boston-area charter schools provided data on student achievement (MCAS scores and course grades), demographics, and school performance (e.g., attendance, high school choices). For the measure of high school choices, BPS high schools were classified as top-tier (referred to in previous evaluations as high quality) or non-top-tier (referred to in previous evaluations as medium or low quality). CS encouraged 8th Grade Academy graduates to attend top-tier schools. In classifying schools, CS considered school size, MCAS scores, percent of graduates attending college or postsecondary training, attendance, dropout rates, grade 9 retention, reputation, and relationship with CS (since an existing partnership would help CS track participants and support them through high school). Test/Assessments: MCAS is a standardized test required by Massachusetts public schools. Results are reported as scaled scores and performance levels, defined as follows: Advanced, has a comprehensive and in-depth understanding of rigorous subject matter and provides sophisticated solutions to complex problems; Proficient, has a solid understanding of challenging subject matter and solves a wide variety of problems; Needs Improvement, has a partial understanding of subject matter and solves some simple problems; and Warning, has a minimal understanding of subject matter and does not solve simple problems. |
| Data Collection Timeframe | Data covered the 2001–2002 academic year (Year 1) through the 2007–2008 academic year (Year 7). |
Findings:
Summative/Outcome Findings
| Academic | Former 8th Grade Academy participants were significantly more likely to select a top-tier high school than were matched nonparticipants (59% vs. 28%, p < .001). Former 8th Grade Academy participants were significantly more likely than nonparticipants to persist in a top-tier high school in grades 11 (67% vs. 49%, p < .05) and 12 (58% vs. 32%, p < .05). In addition, former participants who enrolled in a top-tier high school in 9th grade were more likely to persist in a top-tier high school in 10th grade than were their matched counterparts (85% vs. 75%), although this difference was not significant, Looking across all 4 years, former participants were significantly more likely than nonparticipants to select a top-tier high school and persist in such a school for all 4 years of high school (31% vs. 9%, p < .001). Former participants attended school at significantly higher rates than did matched nonparticipants in every grade of high school. Compared to matched nonparticipants, former participants attended an average of 9 more days in 9th grade (p < .001), 5 more days in 10th grade (p < .05), 13 more days in 11th grade (p < .001), and 9 more days in 12th grade (p < .01). Former 8th Grade Academy participants and matched nonparticipants were suspended at similar rates at all grade levels. No significant difference was found in the suspension rates of former participants and nonparticipants at any grade level. Former participants had significantly higher pass rates in math courses than did matched nonparticipants in grade 9 (74% vs. 65%, p < .05), grade 10 (79% vs. 64%, p < .001), and grade 11 (78% vs. 68%, p < .05). The groups passed at similar rates in grade 12. Former participants had significantly higher rates of earning As and Bs in math courses than did matched nonparticipants in grade 9 (33% vs. 23%, p < .05) and grade 10 (38% vs. 26%, p < .01). Rates were similar in grade 11. Participants outperformed nonparticipants in grade 12, with 33% vs. 23% earning As and Bs, respectively (though this difference was not statistically significant). Former participants passed their ELA courses significantly more often than matched nonparticipants in grade 9 (82% vs. 69%, p < .01) and grade 12 (88% vs. 66%, p < .01). Pass rates were similar between the two groups in grades 10 and 11. While ninth grade participants had a higher percentage of students earning As and Bs in their ELA courses, the difference was not statistically significant. Participants had higher rates of As and Bs in ELA courses in grade 10 (44% vs. 28%, p < .001). While 12th grade participants had a higher percentage of students earning As and Bs than matched nonparticipants (40% vs. 30%), the difference was not statistically significant. Participants and matched nonparticipants passed their math and MCAS examinations at similar rates for both ELA (89% vs. 86%) and math (94% vs. 91). Former participants earned Proficient and Advanced levels on the MCAS at significantly higher rates than their matched peers (53% vs. 44% in math and 50% vs. 41% in ELA, p < .05 for both). A significantly higher percentage of former participants were promoted on time from 9th to 10th grade than were matched nonparticipants (83% vs. 76%, p < .01) On time promotion rates to 11th and 12th grades were not significantly different between the two groups. A significantly higher percentage of former participants were on track to graduate (as defined by passing on time from 9th to 10th grade without having failed a math or ELA course) at the beginning of 10th grade, as compared to their matched peers (63% vs. 51%, p < .01). A significantly higher percentage of former participants graduated on time compared to matched nonparticipants (73% vs. 59%, p < .05). |