You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The New York City (NYC) Department of Youth and Community Development’s (DYCD) Out-of-School Time (OST) Programs for Youth Initiative provides funds to support OST programs for elementary, middle, and high school students across NYC. This initiative is designed to address a broad range of developmental objectives for youth and to serve the needs of NYC’s families and communities.
Start Date September 2005
Scope local
Type afterschool, weekend, summer/vacation
Location urban
Setting public school, community-based organization
Participants elementary through high school students
Number of Sites/Grantees 622 programs (2008)
Number Served 50,618 youth (2005–06); 68,449 youth (2006–07); 81,213 youth (2007–08)
Components Programs operate in NYC schools or community-based organizations (called center-based programs). Trained staff provide a variety of activities including homework assistance, academic support activities, sports/recreational activities, and arts and cultural experiences.

The initiative funds three program options. Option I programs serve elementary through high school students and also includes 15 “Priority Middle Schools” in which OST programs collaborate with state-approved Supplemental Educational Services providers to offer intensive services to youth. Option II programs serve students of any grade level and use private match funds to subsidize at least 30% of their OST budgets. Option III programs collaborate with the NYC Department of Parks and Recreation on three different types of programs: therapeutic recreation and educational services for youth with disabilities, academic support programs, and life skills programs that help youth learn how to manage their finances.
Funding Level over $44 million in 2005–2006; over $117 million in 2008–2009
Funding Sources NYC DYCD
Other DYCD contracted with the Partnership for After School Education (PASE) to provide technical assistance and professional development workshops for OST program staff.


Evaluation

Overview Designed to report on the initiative’s first 3 years, the evaluations examine program characteristics and attendance; participants’ social, emotional, and academic outcomes; and the initiative’s capacity to assist working parents and improve community-level capacities to serve youth during the OST hours.
Evaluators Policy Studies Associates, Inc.
Evaluations Profiled Report on the First Year

Implementation of Programs for High School Youth

Report on the Initiative’s First Three Years
Evaluations Planned Policy Studies Associates will continue to evaluate the program, including an evaluation of the OST Transition to High School program.
Report Availability

Russell, C. A., Reisner, E. R., Pearson, L. M., Afolabi, K. P., Miller, T. D., & Mielke, M. B. (2006). Evaluation of DYCD’s Out-of-School Time Initiative: Report on the first year. Washington, DC: Policy Studies Associates.

Pearson, L. M., Russell, C. A., & Reisner, E. R. (2007). Evaluation of OST programs for youth: Patterns of youth retention in OST programs, 2005–06 to 2006–07. Washington, DC: Policy Studies Associates. Available at: www.policystudies.com/_policystudies.com/files/Year_2_Report.pdf

Russell, C. A., Mielke, M. B., & Reisner, E. R. (2008). Evaluation of the New York City Department of Youth and Community Development Out-of-School Time Programs for Youth Initiative: Results of efforts to increase program quality and scale in year 2. Washington, DC: Policy Studies Associates. Available at: www.nyc.gov/html/dycd/downloads/pdf/ost_evaluation_year_2%20_report.pdf

Russell, C. A., Vile, J. D., Reisner, E. R., Simko, C., Mielke, M. B., & Pechman, E. (2008). Evaluation of the New York City Department of Youth and Community Development Out-of-School Time Programs for Youth Initiative: Implementation of programs for high school youth. Washington, DC: Policy Studies Associates.

Russell, C. A., Mielke, M. B., & Reisner, E. R. (2009). Evidence of program quality and youth outcomes in the DYCD out-of-school time initiative: Report on the initiative’s first three years. Washington, DC: Policy Studies Associates. Available at: www.wallacefoundation.org/KnowledgeCenter/KnowledgeTopics/CurrentAreasofFocus/Out-Of-SchoolLearning/Pages/evidence-of-program-quality-and-youth-outcomes.aspx

Contacts

Evaluation Elizabeth Reisner
Founder and Principal
Policy Studies Associates, Inc.
1718 Connecticut Avenue, NW, Suite 400
Washington, DC 20009
Tel: 202-939-5323
Fax: 202-939-5732
Email: ereisner@policystudies.com
Program Denise Williams
Acting Commissioner
Out of School Time Programs
The City of New York
Department of Youth & Community Development
161 William Street
New York, NY 10038
Tel: 212-676-9845
Email: dwilliams@dycd.nyc.gov
Profile Updated May 9, 2011

Evaluation 3: Report on the Initiative’s First Three Years


Evaluation Description

Evaluation Purpose To explore the associations among program quality, patterns of youth participation, and youth outcomes.
Evaluation Design Quasi-Experimental and Non-Experimental: Evaluators collected data on all 622 sites from an online participant tracking system and surveys of program directors and executive directors of provider organizations. Evaluators administered participant surveys in a random sample of 133 Option I programs. Fifteen Option I “in-depth” sites were randomly selected at which to conduct site visits and interviews with staff, program directors, and youth participants, and to collect additional surveys from staff and parents.

Online program data were collected on 40,584 youth in Option I programs in Year 1; 56,742 youth in Option I programs in Year 2; and 67,524 youth in Option I programs in Year 3. Surveys were completed in spring of each year by
  • 483 program directors from 543 programs in Year 1, 470 program directors of 547 programs in Year 2, and 555 program directors from 630 programs in Year 3
  • 161 of 190 executive directors in Year 1, 148 of 191 executive directors in Year 2, and 169 of 203 executive directors in Year 3
  • 114 staff members at 12 in-depth sites (response rates from 14%– 94% per program) in Year 1, 191 staff members at 13 in-depth sites in Year 2, and 193 staff members at 14 of the 15 in-depth sites in Year 3
  • 283 parents at 12 of the 15 in-depth sites in Year 1 (response rate of 23%), 500 parents at 12 in-depth sites in Year 2, and 450 parents at 13 in-depth sites in Year 3
  • 3,614 youth from 95 programs in Year 1; 5,336 youth from 100 programs in Year 2; and 6,301 youth from 108 programs in Year 3
In each in-depth site, evaluators conducted interviews with the program director, 2–3 staff members, and approximately 5 participants, in each year.  They also interviewed 2–3 parents, mostly in elementary sites.

To analyze the impact of OST participation on academic achievement and school engagement, participants were compared with similar students who did not participate in an OST program. For each OST program, the evaluation identified a primary feeder school (i.e., the school that most of the OST program participants attended). Characteristics of the primary feeder school (e.g., proportion of English language learner [ELL] students, and English language arts [ELA]/math results) were used to identify similar schools without OST or other city-run youth programs from which to select nonparticipants. The evaluation then matched a sample of students from the comparison schools with OST participants, based on grade in school, gender, race, free or reduced-price lunch eligibility, and ELL status. For two high school programs that attracted participants from across the city, similar nonparticipants were selected from a set of schools in the same district without OST programs. This process yielded an evaluation sample of 3,093 participants and 3,093 nonparticipants for the quasi-experimental sample.
Data Collection Methods

Interviews/Focus Groups: Interviews with staff, program directors, and youth gathered information on their perceptions of the program.

Observation: Evaluators used a structured observation instrument to observe program activities during site visits to each of the 15 Option I in-depth study programs.

Secondary Source/Data Review: Evaluators reviewed data from OST Online, DYCD’s program information system, which OST programs use to record and maintain program information, including participant characteristics and OST participation. Evaluators also collected data on all youth in the quasi-experimental sample from Department of Education databases. These included academic performance data on school attendance, math and ELA test scores, and total credits earned.

Surveys/Questionnaires: Surveys of the program director asked about program goals, activities, and schedules; staff recruitment and qualifications; participant outreach, recruitment, needs, and preferences; and linkages with schools, communities, and families.

Surveys of the executive directors asked about how the OST program influenced the fulfillment of the provider organizations’ core missions, how OST programs linked to other services delivered by the provider organization, and about the cost and funding of specified program elements.

Participant surveys asked about such issues as youth participants’ perceptions of the program including its benefits and their experiences in the program.

Program staff surveys asked about staff’s background, qualifications, and experience working in the program.

Parent surveys asked about program perceptions, perceived benefits, and how and whether the program met parents’ needs.

Tests/Assessments: Through observations, evaluators recorded and assessed: (a) staff instructional strategies that foster mastery (e.g., staff communicate goals, purposes, and expectations); (b) the quality of the activity content and structure (e.g., the activity requires analytic thinking); and (c) positive relationships displayed in activities (e.g., youth respect one another). Indicators within these scales were rated from 1 to 7, with higher numbers indicating greater presence of the desired features.

Evaluators constructed various scales from participant survey questions, including scales tapping youth participants’ sense of belonging in the program, engagement in pro-social behaviors such as helping others, interacting with peers and staff, academic motivation, and perceptions of academic benefits from the program.

Based on all correlations between program features (as measured by director surveys, the OST Online system, and aggregated youth surveys) and youth outcomes over the three years, evaluators created a program quality index as a tool for assessing the overall quality of an OST program. Final components of the program quality index included exposure to new experiences, youth interactions with peers, youth interactions with staff, wide mix of staff, presence of a master teacher, and presence of a parent liaison.

Evaluators also collected Regents exam test score data from the New York City Department of Education. These exams are mandatory state-level subject-area tests administered to all students in the state of New York in grades 3–12.
Data Collection Timeframe Data were collected in Year 1 (2005–2006), Year 2 (2006–2007), and Year 3 (2007–2008).


Findings:
Formative/Process Findings

Activity Implementation Nearly all elementary and middle school OST programs offered academic enhancement, arts and culture, and recreational activities. About two thirds of these programs offered life skills activities, and about half offered community-building activities. Consistent with the age ranges they serve, relatively few elementary and middle school OST programs offered any career or work activities (15% and 24%, respectively).

High school OST programs often focused on a narrow set of activities or topic areas. Reflecting that focus, the activity patterns in high school programs differed from those of elementary and middle school programs: while more than two thirds of high school programs offered academic enhancement, arts and culture, and recreational activities, only about half of these programs offered life skills, community-building, and career or work activities.

Overall, participants reported a high degree of exposure to new experiences through the program (average score of 3.20 out of 4 on the youth survey scale). Elementary school youth were somewhat more likely to report a high level of participation in new experiences (3.34 out of 4) than were middle school youth (3.22) and high school youth (3.27).

On average, observed activities rated highly on clear activity goals (5.07 out of 7) and the extent to which activities developed personal and social skills (4.21 out of 7). In contrast, observations revealed less evidence of activity quality as defined by the extent to which the activity built on skills and content previously learned (3.60, out of 7) or engaged youth in active, hands-on learning experiences (1.75, out of 7).

Parents especially valued the academic support features of OST programs: 47% cited homework help as the most important activity in the OST program, and an additional 26% cited academic enrichment as the most important activity.
Parent/Community Involvement Nearly all program directors (91%) had conversations with parents over the phone at least a few times a month; 83% met in person with parents with similar frequency.

Programs relied on family or parent liaisons to engage families and encourage greater participation: 45% of programs employed someone for this role.
Program Context/ Infrastructure The initiative expanded the number of programs offering OST services over the 3 years of the evaluation, with the addition of 111 new elementary school OST programs in the third year.

Participants reported an average score of 3.38 out of 4 on the youth survey scale of “sense of belonging” in the program. In particular, more than two-thirds of participants “agreed a lot” that they “feel safe” in the program (68%) and 58% said that the program was a “good place to hang out.” High school students were especially likely to report a strong sense of program connection (with an average score of 3.48, compared to 3.32 for elementary school participants and 3.38 for middle school participants).

Analyses of youth reports revealed positive relationships between the extent to which a program exposed youth to new experiences and of the youths’ sense of belonging in the OST program (p < .05).

Measures of a supportive OST environment, including youth reports of interactions with their peers and with staff members in the program, were positively and significantly correlated with youth reports of their sense of belonging (p < .05).

Analyses found positive and significant correlations between overall program quality and aggregate youth reports of their sense of belonging in the program (p < .05).
Program–School Linkages Program directors reported communicating with school administrators or staff at least monthly about the needs or progress of individual students (61%), issues related to sharing classroom space (56%), homework assignments (56%), and student discipline policies (50%).

In surveys, 61% of program directors said that receiving responses to requests to coordinate services or resources with school staff was not a challenge; only 6% reported that this issue was a major challenge.
Recruitment/ Participation The OST initiative grew over its first 3 years to serve increasingly large numbers of youth each year. In Year 1, 50,618 youth enrolled in OST programs throughout the city, including 40,584 participants in the Option I programs. In Year 2, this number had increased to 68,449 participants overall with 56,742 in Option I programs, and by Year 3 the initiative had grown to serve 81,213 youth, including 67,524 Option I program participants.

A total of 6,371 youth participated in all 3 years of OST school-year programming. Almost 22,000 youth participated in 2 years of OST programming, either in Years 1 and 2 or in Years 2 and 3, while 102,837 participated in a single year of programming (Year 1, 2, or 3). Approximately 22,000 youth participated only in summer programming.

In the third year of the OST initiative, programs on average exceeded their targeted enrollment levels (measured by the number of slots available for participants as established in the program’s contract with DYCD). Option I programs had a target enrollment overall of approximately 63,000 youth; programs actually served a total of about 64,500 students during the school year. Even so, some individual programs could not meet their targeted enrollment: 31% of elementary school youth programs did not meet or exceed their enrollment targets, nor did 30% of middle school youth programs or 41% of high school youth programs.

On average, elementary school youth participants attended 377 hours of OST programs during the year, compared to the 432 hours they were expected to attend. This attendance represents an average of 87% of targeted hours, exceeding average participation rates of 72% and 83% attained in Year 1 and Year 2, respectively. Middle school youth participants as a group achieved their targeted number of hours of participation: on average, middle school participants attended 218 hours of the 216 hours expected at the middle school level. High school participants also exceeded their target of 76 hours of participation, attending on average 92 hours in Year 3.

A program’s mean rating of exposing youth to new experiences was negatively correlated with mean hours of participation of individual youth in OST; this correlation was significant (p < .05).

The breadth of content in OST programs, measured by the number of different activity types offered, was positively and significantly associated (p < .05) with the total number of hours of youth participation.

Measures of communication with schools and with parents were positively and significantly associated (p < .05) with the number of hours of youth participation in OST programming.

Analyses found a positive and significant relationship between the program quality index and whether the program succeeded in meeting its targeted enrollment level (p < .05).

Parents’ reports of their reasons for enrolling their child in the OST program reflected an emphasis on seeking academic support: 76% believed the program would help their child do better in school, and 72% wanted their child to get help with homework. Seventy-five percent of parents also said that they enrolled their child in an OST program to provide them with the opportunity to participate in new activities.
Satisfaction Youth reported a high level of social interactions within the program (average score of 3.32, out of 4), although both middle and high school youth responded somewhat more positively than did elementary school youth participants (3.35 and 3.37 vs. to 3.28).
 
About three quarters of parents rated the OST program that their child attended as either excellent (43%) or very good (33%).
Staffing/Training The majority of programs (82%) hired college students, and 69% of programs hired at least one professional specialist (e.g., a professional artist, coach, or dancer). In addition, 63% of programs had at least one certified teacher on staff, and 61% hired teen staff members. Fifty percent of OST program directors reported that a staff member was assigned to be a master teacher or educational coordinator within the program.

OST programs were strategic in the roles they assigned to certified teachers and specialists within the programs. Certified teachers were employed mainly to provide academic support to programs by leading academic activities (72% of programs) and tutoring (71%). Specialists were hired primarily for non-academic enrichment activities such as arts and sports (88%). College students, in contrast, played roles across program activities, including tutoring youth (88%) and assisting with enrichment (82%) and academic (78%) activities.

OST participants responded with high ratings on a scale measuring the quality of their interactions with program staff (average score of 3.35 out of 4), with high school youth responding somewhat more positively than either elementary or middle school youth (3.46 for high school youth vs. 3.31 for both elementary and middle school youth).

Program directors reported that their staff received professional development through staff meetings at the program (86%), internal staff orientations (66%), and off-site workshops (62%).


Summative/Outcome Findings

Academic Participants reported academic benefits on a survey scale (average score of 3.06 out of 4). The most common academic benefit reported by participants was that the program helped them to finish their homework more often (54% “agreed a lot”).

No significant differences were found between participants and matched nonparticipants on the measures of academic performance that are maintained by the Department of Education (attendance, test scores, or credits earned). Consistent with citywide trends, both groups showed small improvements in performance over the course of the OST initiative, with no significant differences in the size of the gains between the two groups.

For high school youth in the sampled sites, analyses found no significant differences between the program and comparison groups on the cumulative number of course credits accrued after each year of OST participation.

Analyses found no significant differences in the number of New York State Regents exams that participants and matched nonparticipants had taken and passed.

Analyses of school attendance patterns of participants and matched nonparticipants found no significant differences in attendance changes over time.

Analyses revealed positive and significant relationships (p < .05) between youth reports of the extent to which a program exposed youth to new experiences and their perceptions of academic benefits and rates of school attendance.

The breadth of the type of content in OST programs, measured by the number of different activity types offered, was positively and significantly associated (p < .05) with the number of credits earned by high school participants.

Measures of a supportive OST environment, including average youth reports of their interactions with their peers and with staff members in the program, were positively and significantly correlated with youth reports of the program’s academic benefits (p < .05).

Analyses found positive correlations between overall program quality and youth’s academic motivation and perceptions of academic benefits.
Family Across all surveyed parents, 74% agreed that the program made it easier for them to keep their job, and 73% agreed that they missed less work than they had previously because their children attended the OST program. In addition, 71% of parents reported that they were able to work more hours because their children were in the program.
Systemic More than half of executive directors reported that the initiative increased “somewhat” or “to a great extent” the organization’s capacity to reach out to serve more youth and families (83%); provide staff training and technical assistance (73%); partner with a public school (71%), cultural organizations (65%), or city agencies (63%); offer programming on weekends and holidays (59%); and provide a career ladder for OST staff (57%).
Youth Development Analyses found a positive and significant correlation between overall program quality and youth’s engagement in pro-social behaviors as reported on the youth survey (p < .05).

 

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project