Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of After School Achievement Program

Program Description

Overview	The After School Achievement Program (ASAP) was founded in 1997 by the City of Houston, with the support of the Joint City/County Commission of Children and other community leaders, to provide after school programming to Houston's children and youth. The ASAP program has six goals: (1) to reduce crime committed by and against juveniles, (2) to prevent delinquency, (3) to provide a safe, supervised place for youth, (4) to provide academic enhancement and enrichment, (5) to promote school attendance and discourage school drop out, and (6) to motivate youth to develop good citizenship. The program has expanded from 7 sites in 1997 to 95 sites in 2000–2001.
Start Date	1997
Scope	local
Type	after school
Location	urban

Setting	public schools, community-based organizations, faith-based organizations
Participants	elementary through middle school students
Number of Sites/Grantees	95 sites, each receiving a grant in the range of $12,650 to $40,000
Number Served	11,649 children/month (2000–2001)
Components	All ASAP grantees must provide four major program components to students: 1. Academic enrichment (e.g., homework assistance, academic remediation, career awareness, and technology-based activities) 2. Personal skills development (e.g., leadership/good citizenship, team building, conflict management/violence prevention, and volunteerism) 3. Enrichment activities (e.g., recreation/sports, field trips, fine arts, and creative activities) 4. Community involvement (e.g., senior home visitations, community restoration projects, community/parental involvement, continuous feedback from teachers/parents, and adult/youth volunteerism)
Funding Level	$2.3 million (2000–2001). All grantees were required to provide a “dollar for dollar” match for additional operational expenses.
Funding Sources	City of Houston

Evaluation

Overview	Evaluations of the ASAP program were conducted annually for four years, from 1998 through 2001. Smith et al., of the University of Houston's College of Education, carried out all evaluations, focusing on ASAP attendance, program availability and variety, crime and academic impacts, and perceptions of quality by participants, parents, and Facilitators/teachers.
Evaluators	Dennis W. Smith et al., University of Houston, College of Education
Evaluations Profiled	After School Achievement Program (ASAP), 1998 Great Cities Do Great Things: Year 2 Evaluation of the After School Achievement Program (ASAP), 1999 Terrific Children: A City-Wide Program to Support Communities Year 3 Evaluation of the After School Achievement Program (ASAP), 2000 Shaping our Children's Future: Keeping a Promise in Houston Communities Year 4 Evaluation of the Mayor's After-School Achievement Program (ASAP), 2001
Evaluations Planned	not available
Report Availability	Smith, D. W., Zhang, J. J., Cheng, E. Y., & Lam, E. T. C. (1998). After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., Zhang, J. J., Rodriquez, A., Haynes, M., & Brimer, J. (1999). Great cities do great things: Year 2 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., Zhang, J. J., Brimer, J., & Rodriquez, A. (2000). Terrific children: A city-wide program to support communities Year 3 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., & Zhang, J. J. (2001). Shaping our children's future: Keeping a promise in Houston communities Year 4 evaluation of the Mayor's After School Achievement Program (ASAP). Houston, TX: University of Houston.

Contacts

Evaluation	Dennis W. Smith Email: dwsmith@uh.edu
Program	Jennifer Brimer Sr. Community Liaison Coordinator After School Achievement Program City of Houston P.O. Box 1562 Houston, TX Tel: 713-437-6981 Email: jenniferbrimer@cityofhouston.net
Profile Updated	January 27, 2003

Evaluation 2: Great Cities Do Great Things: Year 2 Evaluation of the After School Achievement Program (ASAP), 1999

Evaluation Description

Evaluation Purpose	To examine the functioning and quality of the After School Achievement Program during the 1998–1999 school year, with a focus on impacts on academic achievement and violence prevention.
Evaluation Design	Non-Experimental: Data collection occurred at 58 of the 68 existing program sites. Twenty sites were randomly selected for more intensive study of student, teacher, and parent perceptions. Of these, 17 provided usable student data, 17 provided usable parent data, and all 20 provided usable teacher data. The students that the teachers evaluated were selected by the teachers.
Data Collection Methods	Document Review: Documents reviewed for the evaluation included ASAP records of program attendance, participants, budget, and site curricular activities, school reports of attendance and grades, and archived police documents regarding crime data for local precincts. The ASAP records and school reports were collected for all 58 participating sites. The police reports were collected for only 20 sites Surveys/Questionnaires: Students, parents, and teachers were surveyed about their perceptions and opinions of program activities at their sites. All surveys were validated during Year 1 of the ASAP evaluation, including tests of content and construct, and revised for use in the 1998–1999 evaluation. One-thousand-twenty-three students at 17 program sites were surveyed about their perceptions of the program and their intentions for future participation using a 27-item Likert 3-scale survey for all but two questions where satisfied, dissatisfied, and neutral were represented by smiling, neutral, or frowning faces. Five-hundred-two parents from 17 sites were surveyed about their perceptions of the quality of the program, their intentions to send their child in the future, and their suggestions using a 32-item Likert 5- and 3-scale survey. Teachers from 20 sites were surveyed about 182 student ASAP participants to determine teachers' perceptions of individual student improvements in the cognitive, affective, and social domains as a result of ASAP participation. The teachers selected which students to evaluate. The teacher surveys were comprised of 18-items in Likert 5-scale format.
Data Collection Timeframe	Data collection occurred between September 1998 and June 1999.

Findings:
Formative/Process Findings

Activity Implementation	In general, ASAP programs followed the City of Houston program guidelines with respect to activities and offered activities that cut across the four component areas of academic enhancement, personal skills, enrichment activity, and community involvement. Two-hundred-fifty different activities were cumulatively offered by sites in the areas of academics, professional development, life skills, arts and crafts, sport/exercise, domestics, games, field trips, and social activity. The most commonly adopted activities include: arts and crafts, computer instruction, homework/tutorial, Texas Assessment of Academic Skills (TAAS) practice, team building, leadership, recreation and sports, and community collaboration. The 10 most frequent “academic enhancement” activities, significantly (p<.05) more common than other activities, in order, are: homework assistance, computer classes, TAAS, reading, math workshop, academic remediation, career awareness, technology activities, science, and group story time. The 10 most frequent “personal skills” activities, significantly (p<.05) more common than other activities, in order, are: team building games, good citizenship, leadership, conflict management/violence prevention, volunteerism, cooperative learning, following directions, accident prevention, character education, and drug free school club. The 10 most frequent “enrichment” activities, significantly (p<.05) more common than other activities, in order (from most to least frequent), are: recreation and sports, arts and crafts, dance, basketball, field trip, soccer, gymnastics music, fine arts, and aerobics. The 10 most frequent “community involvement” activities, significantly (p<.05) more common than other activities, in order (from most to least frequent), are: volunteer, parents volunteer, collaboration (teachers, parents, school), continuous feedback, community/parent contribution, read write now, adult computer class, multicultural craft, parent conferences, and teacher aid volunteer.
Costs/Revenues	Ninety-five percent of programs operate within the approved budget. The basic program cost was $200 per participant in addition to some supplementary money from the Child Safety fund.
Program Context/Infrastructure	Over 75% of program sites adopted a five-day per week schedule and most had operating hours of 2:30pm to 6:30pm. Approximately 90% of programs served a snack. The YMCA of Greater Houston or Chapter I/Title I were primary providers of snacks. All program sites followed program reporting requirements and approximately two thirds indicated that their program reports were complete and accurate. Fifty-nine percent of student survey respondents indicated that if there were no after school program they would be at home; 13% said they would be with a friend; 9% said they would be with a relative; 7% said that they would be on the street; 7% said they would be at the park; 3% answered “other.” These findings were statistically significant (p<.000). There were no findings about the level of supervision at any of these sites. Student survey respondents indicated that if there were no after school program they would be engaged in the following activities with the correspondent frequencies: 28% would be watching TV, 26% would be doing homework, 23% would be playing, 12% would be hanging out, 7 % would be doing nothing, and 3% would be doing some other activity. These findings were statistically significant (p<.000). There were no findings about the level of supervision during these activities. 1998–1999 ASAP sites were located in police precincts with elevated crime rates-more than 55% of crime victims and suspects according to the 1997–1998 data were in ASAP located police precincts.
Recruitment/Participation	Based on responses from 58 of 68 total sites, there were 7,700 participants in ASAP during the 1998–1999 school year. Participants were 51% Hispanic and 41% African-American with these ethnicities significantly (p<.000) more common than others. Approximately 5% were Asian/Pacific-Islander, 3% were Anglo/Caucasian, .01% were American-Indian, and .01% were “other.” Seventy-eight percent of participants receive free or reduced-price lunch. Forty-one percent of participants were from homes run by single mothers. Sixty percent of participants attended the program on a daily basis. A majority of program sites began operation between September and November of 1998 and ran until May 1999. Most of the programs had enrollment capacities of either 50–99 children (47%) or 100–199 (33%). Three-quarters of sites were able to reach a minimum of 60 participants. Of the 25% who were not, a substantial proportion cited reporting difficulties rather than barriers to participation.
Satisfaction	Ninety percent of all student survey respondents, who comprised 46% of all students participating in the program at the surveyed schools, indicated that they liked the ASAP program, 6% were unsure, and 3% indicated that they did not. There were significantly (p<.000) more students who felt positively about the program than students who did not. Seventy-eight percent of all student survey respondents indicated that they wanted to attend the program next year, 14% were unsure, and 9% indicated that they did not want to. There were significantly (p<.000) more students who wanted to attend the program next year than students who did not. The majority (ranging from 78% to 92%) of student survey respondents indicated that they were satisfied regarding the remaining 25 survey items. Statistical tests revealed that there were significantly (p<.05) more students who felt satisfied than students who did not. Of the 520 parents from 17 randomly selected schools who responded to the parent survey, 50% of the eligible parents, 77% gave the ASAP program an overall rating of “excellent.” In addition, parents predominantly gave the most positive response on the remaining Likert 4-scale survey items, indicating widespread satisfaction with the program. Mean scores on all parent survey items were found to be significantly (p<.05) greater than neutral (2.5). Ninety-eight percent of parents indicated on the parent survey that they intended to enroll their child in the ASAP program the following year. This finding was statistically significant (p<.05). Parents' positive perceptions of the program fell across all five content areas including general operation, leadership quality, program materials, future intentions, and personal needs. The evaluators gave the ASAP program an “A” in the domain of “student evaluation” since the program consistently yielded high student ratings. The evaluators gave the ASAP program an “A” in the domain of “parent evaluation” since the parents were extremely positive about the ASAP program.
Staffing/Training	Each program site had an average of 11 paid staff members and 6 volunteer staff members. The paid staff averaged less than one hour/week/person while the volunteer staff averaged more than four hours/week/person.

Summative/Outcome Findings

Academic	Each program site had over 80 students, usually more than 60% of all participants, who improved their academic grades, school attendance, and school behaviors. The percentage of students who improved their grades on average was significantly (p <.05) greater than 50%.
Prevention	1997–1998 and 1998–1999 data from the Houston police precincts showed a dramatic reduction in youth victimization rates in precincts with ASAP programs compared with a much smaller reduction in non-ASAP precincts. During 1998–1999, the decline in serious crimes involving a juvenile victim was 15% in ASAP districts and 12% for all crimes involving a juvenile victim. The decline is more than three times that of non-ASAP precincts for serious crimes against juveniles and more than 1.5 times that of non-ASAP precincts for all youth victimizations. The difference between ASAP and non-ASAP precinct crime reductions was statistically significant (p<.05) both for the total of all violent crimes where juveniles were victims and the total of all crimes in which juveniles were victims. 1997–1998 and 1998–1999 data from Houston police precincts showed a marked decline in crime where youths were suspects in the districts with ASAP programs compared with a much smaller reduction in non-ASAP districts. During 1998–1999, ASAP precincts experienced a 13% decline in serious crimes involving a youth suspect and total crimes involving a youth suspect. The ASAP precinct decline was more than 15 times that of non-ASAP precincts for serious crimes with a youth suspect and more than 1.2 times the non-ASAP reduction in total youth crime suspects. The proportional reduction in violent crimes with youth suspects and all crimes with youth suspects between ASAP and non-ASAP precincts was statistically significant (p<.05). For every $2 per month spent on the children in the program, a one percent decline in violent crimes and suspects was realized in police precincts with ASAP sites compared to precincts without ASAP sites.
Youth Development	Teachers evaluating 182 students from 20 randomly selected sites, approximately 10% of students in the ASAP programs at their schools, predominantly indicated that students had made progress in the cognitive, affective, and behavioral areas as a result of participation in the ASAP program. Consistently the mean scores on the 18 survey items in Likert 5-scale format were significantly (p<.05) greater than neutral (3). The highest mean items on the teacher evaluations of students, both having a mean of 4.6/5, were related to children having adults who care about them in the after school program.

A Profile of the Evaluation of After School Achievement Program

Profile Information