Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of After School Achievement Program

Program Description

Overview	The After School Achievement Program (ASAP) was founded in 1997 by the City of Houston, with the support of the Joint City/County Commission of Children and other community leaders, to provide after school programming to Houston's children and youth. The ASAP program has six goals: (1) to reduce crime committed by and against juveniles, (2) to prevent delinquency, (3) to provide a safe, supervised place for youth, (4) to provide academic enhancement and enrichment, (5) to promote school attendance and discourage school drop out, and (6) to motivate youth to develop good citizenship. The program has expanded from 7 sites in 1997 to 95 sites in 2000–2001.
Start Date	1997
Scope	local
Type	after school
Location	urban

Setting	public schools, community-based organizations, faith-based organizations
Participants	elementary through middle school students
Number of Sites/Grantees	95 sites, each receiving a grant in the range of $12,650 to $40,000
Number Served	11,649 children/month (2000–2001)
Components	All ASAP grantees must provide four major program components to students: 1. Academic enrichment (e.g., homework assistance, academic remediation, career awareness, and technology-based activities) 2. Personal skills development (e.g., leadership/good citizenship, team building, conflict management/violence prevention, and volunteerism) 3. Enrichment activities (e.g., recreation/sports, field trips, fine arts, and creative activities) 4. Community involvement (e.g., senior home visitations, community restoration projects, community/parental involvement, continuous feedback from teachers/parents, and adult/youth volunteerism)
Funding Level	$2.3 million (2000–2001). All grantees were required to provide a “dollar for dollar” match for additional operational expenses.
Funding Sources	City of Houston

Evaluation

Overview	Evaluations of the ASAP program were conducted annually for four years, from 1998 through 2001. Smith et al., of the University of Houston's College of Education, carried out all evaluations, focusing on ASAP attendance, program availability and variety, crime and academic impacts, and perceptions of quality by participants, parents, and Facilitators/teachers.
Evaluators	Dennis W. Smith et al., University of Houston, College of Education
Evaluations Profiled	After School Achievement Program (ASAP), 1998 Great Cities Do Great Things: Year 2 Evaluation of the After School Achievement Program (ASAP), 1999 Terrific Children: A City-Wide Program to Support Communities Year 3 Evaluation of the After School Achievement Program (ASAP), 2000 Shaping our Children's Future: Keeping a Promise in Houston Communities Year 4 Evaluation of the Mayor's After-School Achievement Program (ASAP), 2001
Evaluations Planned	not available
Report Availability	Smith, D. W., Zhang, J. J., Cheng, E. Y., & Lam, E. T. C. (1998). After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., Zhang, J. J., Rodriquez, A., Haynes, M., & Brimer, J. (1999). Great cities do great things: Year 2 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., Zhang, J. J., Brimer, J., & Rodriquez, A. (2000). Terrific children: A city-wide program to support communities Year 3 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., & Zhang, J. J. (2001). Shaping our children's future: Keeping a promise in Houston communities Year 4 evaluation of the Mayor's After School Achievement Program (ASAP). Houston, TX: University of Houston.

Contacts

Evaluation	Dennis W. Smith Email: dwsmith@uh.edu
Program	Jennifer Brimer Sr. Community Liaison Coordinator After School Achievement Program City of Houston P.O. Box 1562 Houston, TX Tel: 713-437-6981 Email: jenniferbrimer@cityofhouston.net
Profile Updated	January 27, 2003

Evaluation 3: Terrific Children: A City-Wide Program to Support Communities Year 3 Evaluation of the After School Achievement Program (ASAP), 2000

Evaluation Description

Evaluation Purpose	To explore the function and quality of the 1999–2000 ASAP program, with a focus ASAP's impact on academic achievement and violence prevention. Additionally, the evaluation looked at trends established over the three-year period of the program's operation.
Evaluation Design	Non-Experimental: Data were collected from 111 sites and about 11,786 children. Forty sites, and 3066 children and 1,695 parents at those sites, were randomly selected for more intensive study of parent and student perceptions. Crime data were gathered from 103 ASAP sites and local beats.
Data Collection Methods	Document Review: Documents reviewed for the evaluation included ASAP records of program attendance and site curricular activities, school reports of attendance and grades, and archived police documents regarding crime data for local precincts. All 111 participating sites provided the ASAP and school documents, but only 103 provided the police documents. Data was collected from September 1999 through June 2000. Interviews/Focus Groups: Semi-structured interviews were conducted with 10 facilitators, 46 parents, and 55 children about their perceptions of the value of the ASAP. Interviews were conducted in July and August 2000. Surveys/Questionnaires: Students, parents, and teachers were surveyed about their perceptions and opinions of program activities at their sites. All surveys were validated during Year 1 of the ASAP evaluation, including tests of content and construct, revised for use in the 1998–1999 evaluation, and tested and validated again in 1999–2000. Three-thousand-sixty-six children from 40 sites responded to the Youth Survey, which consisted of 30 items in Likert scale format and focused on their perceptions of the program and their intention to participate next year. One-thousand-six-hundred-ninety-five parents from 40 sites were surveyed about their perceptions of the quality of the ASAP program, their intention to send their children to ASAP in the future, and how their children's after school needs would be met in the absence of ASAP with a 33-item Likert scale format instrument. Both parent and student surveys were available in Spanish and English. Five-hundred-three Facilitators (teachers) from 103 sites responded to an 18-item Likert scale survey about their students' improvements in the cognitive, affective, and social domains as a result of participation in ASAP.
Data Collection Timeframe	Data were collected from September 1999 through June 2000.

Findings:
Formative/Process Findings

Activity Implementation	On average, there were 61 children in each activity offering. Over all three years of program operation, ASAP programs consistently followed the stated curriculum and its four objectives (academic enhancement, personal skills, enrichment activity, and community involvement). Program activities of ASAP sites generally followed the program guidelines in that there were activities in all four domains: academic enhancement, personal skills, enrichment activity, and community involvement. Activities that fell into these four domains had significantly (p<.05) greater frequencies than other activities. Interviews with children revealed the theme that they are engaged in personally rewarding activities.
Costs/Revenues	The per child annual cost was about $225, 10% more than in 1998–1999, but still approximately equal in cost to one month of privately arranged care. Variables used to define cost effectiveness include the projected per site funding by the City of Houston, the number of paid personnel, a constant personnel/operational expense projection, and the average number of children served. Total cost was $1.63/day/child for ASAP. The operational (excluding personnel) costs of ASAP were $.41/day/child. The funded personnel costs were about $9.25/hour/worker and a cost per child of $1.22/child/day. Volunteers and in-kind contributions from each site supplemented the ASAP budget.
Program Context/Infrastructure	The evaluators sought to understand what ASAP participants would be doing in the absence of the program. Fifty-three percent of student survey respondents indicated that they would be at home if there were no after school program; 7.2%, or 221 children, responded that they would be on the street. If there were no after school program, about half of student survey respondents indicated that they would be doing homework or playing; 25% indicated that they would be doing nothing or hanging out; and 20% indicated that they would be watching TV. Approximately 50% of student survey respondents indicated that they would have adult supervision if there were no after school program, while 20% said that they would have no one to watch them. Forty percent of parent survey respondents indicated that if there were no after school program, their children would be supervised by a relative or sitter; 30% indicated that their children would be by themselves; and 20% indicated that their children would be watched by a sibling. Interviews with children revealed the theme that ASAP participants would be home alone without the program. Programs operated for an average of 16 days per month and 3 hours per day. Significant (p<.05) differences existed in terms of program operation variables among different types of ASAP program sites. Nonprofit organizations that hosted ASAP sites offered more days and hours of operation than Houston Independent School District (HISD) sites, which in turn, offered more days and hours monthly than other public school sites. However, nonprofit and HISD sites offered comparable ASAP hours per day, which were significantly (p<.05) greater than other public school site hours/day. No significant (p<.05) differences were found in academic outcomes between the three types of program sites (HISD, other public schools, and nonprofit organizations). HISD sites had significantly (p<.05) more participants per activity than other public schools who, in turn, had more participants per activity than nonprofit ASAP sites. Generally speaking, the number of participants in different program activities were similar for the three different host organizations (nonprofits, HISD schools, and other public schools). However, arts/crafts activities had significantly (p<.05) more participants per at HISD sites than at nonprofits which had more participants in arts/crafts than other public school ASAP sites. For athletics/recreation activities, HISD and other public school ASAP sites had significantly (p<.05) more participants per activity than nonprofits did. And for tutorial/homework, other public school sites had significantly (p<.05) more children per session than HISD, which had more than nonprofit organizations hosting ASAP. There was no significant difference between the enrollments of African-American and American-Indian youth served in each of the three program venue types. Significant (p<.05) differences did exist in terms of Asian, white, and Hispanic enrollment: more Hispanics were served in HISD school sites, more Asians were served in other public school sites, and more whites were served in HISD and other public school sites. No significant (p<.05) differences were found between the three types of programs in terms of new enrollments or the number of youth from female-headed households. Significant (p<.05) differences between the three types of sites were found in the total monthly enrollment, overall program withdrawal, youth receiving fee or reduced-price lunch, and average daily attendance. On all of these variables, HISD school sites were greater than other public school sites, which were, in turn, greater than nonprofit sites.
Recruitment/Participation	Over three years of program participation, ASAP consistently served Hispanic and African-American youth, low-income youth, and youth from single parent families. Cumulatively, Hispanic youth represented 50% of all participants and African-American youth represented 41%; 80% of all participants received free or reduced-price lunch; and youth from female-headed single parent families comprise 40% of all participants. The number of ASAP sites and participants increased each of its three years of operation from 11 sites and 1,500 participants in 1998, to 68 sites with 7,700 participants in 1998–1999, to 111 sites with over 12,000 children in 1999–2000. Approximately 11,786 children were served in the ASAP program during the 1999–2000 school year each month based on reports from 111 of the 112 sites, a quarter of which were new enrollments. The monthly average was approximately 100 children per site. Seventy-eight percent of participants receive free or reduced-price lunch, while 88% qualify. Thirty-seven percent of participants are from homes with single mothers. An average of 69% attended ASAP on a daily basis. Forty-eight percent of ASAP participants were Hispanic; 45% of ASAP participants were African-American; 4.5% were Anglo; and 2.1% were Asian. Statistical tests reveal that the percentages of African Americans and Hispanics were significantly (p<.000) greater than the proportions of Asians/Pacific Islanders, Anglo/Caucasians, American Indians, and other ethnic groups. There were approximately equal numbers of male and female children. Fifty-five percent of parent survey respondents had one child in ASAP and 45% had two or three children in the program. Ninety-seven percent of children were between the ages of 5 and 12 and in the first through eighth grades. About 46% of parent survey respondents considered their incomes inadequate or uncomfortable. Most of the programs met the requirement of having 60+ participants on a daily basis. Sites with higher numbers of youth from female-headed households and/or are receiving free or reduced-price lunch had higher numbers of participants. The number of youth receiving free or reduced-price lunch and the number of Hispanic youth were positively related to the number of new monthly enrollments, testifying to the value of such programs in Hispanic communities and the need for outreach into such communities to recruit and retain participants. The number of youth receiving free or reduced-price lunch, the number of youth from female-headed households, the number of Hispanic youth, and the number of white youth were all positively associated with the number of youth withdrawing from the program each month, so evaluators conclude that program retention efforts should focus on youth with these backgrounds.
Satisfaction	Over three years of program operation, there was no significant change in teacher perceptions of the program's impact on participating students or student and parents' satisfaction with the program. All were consistently positive. Seventy-seven percent of student survey respondents indicated that they “definitely” liked ASAP, while 18% said they “kind of” liked ASAP, and the remaining 5% said that they did “not at all” like ASAP. Significantly (p<.000) more students felt positively about the program than negatively. Seventy-one percent of student survey respondents indicated that they “definitely” wanted to attend ASAP next year, while 17% said they “kind of” wanted to attend ASAP next year, and the remaining 12% said that they wanted “not at all” to attend ASAP next year. Significantly (p<.000) more students wanted to attend ASAP next year than did not. There were significantly (p<.05) more positive student survey responses than negative ones, indicating overall satisfaction with ASAP. In addition, on every survey item, a majority of students gave the most positive response. Parent survey respondents gave high ratings to the ASAP program, with the majority giving the program the most positive rating on almost all of the survey items. All mean scores were significantly (p<.05) more positive than neutral. Parents' perception of the functioning and offerings of the ASAP program were positively and significantly (p<.05) associated with the number of their children in ASAP, i.e., the more children of theirs in the program, the more positively they felt about the functioning and offerings of ASAP. Parents' perceptions of program functioning were significantly (p<.05) negatively related to the age of their participating child. Parents of younger children were relatively more satisfied with the program than parents of older children. Parents' perception of the program's leadership were significantly (p<.05) negatively related to family income which means that parents of low-income children were relatively more satisfied with the program than parents of higher income children. Parents' satisfaction with program leadership and operation was found to be greater among parents of white and African-American children and lesser among parents of Asian-American and American-Indian children. American-Indian parents were satisfied with program leadership, but not program operation. Parents who indicated that in the absence of ASAP they themselves, a relative, or a sitter would care for children were more satisfied with program functioning. Parents answering “a relative” or “themselves” were also more satisfied with program leadership. Parents answering “by themselves” or “by another organization” were more satisfied with program operation. These findings may suggest that ASAP was more satisfactory to those who had the most needs. Parent evaluation of ASAP was found to be unrelated (p<.05) to children's gender or their qualification for free or reduced-price lunch.

Summative/Outcome Findings

Academic	Data from the 1998–1999 and 1999–2000 evaluations revealed that ASAP programs consistently had at least 50% of program participants who made improvement in academic performance, school attendance, and school behaviors, respectively. On average, each program had more than 50 students, or 50% of ASAP participants, who improved school grades, school attendance, and school behavior. The percentage to have improved in this way was found to be significantly (p<.05) greater than 25% and not significantly (p<.05) different from 50%. Program enrollment size, number of new enrollments, number of withdraws, and daily attendance was not found to be significantly (p<.05) related to youth school grades, attendance, and behaviors, indicating that program size did not affect quality of the program.
Community Development	Interviews with facilitators revealed the theme that ASAP enhances the quality of life in communities. Interviews with parents revealed the theme that ASAP creates support networks within the community.
Prevention	Out of a total of 117 police beats, 57 had one or more ASAP programs, ranging from one to six. There were no significant (p<.05) differences between beats with after school programs and those without for all crime variables, including number of violent suspects, number of nonviolent suspects, number of violent victims, and number of nonviolent victims. Considering that crime histories were different between the beats with and without the programs, juvenile crime data for the 1997–1998 and 1998–1999 years were used as covariates and their effects were partialled out in the analyses. Possible reasons for non-significant crime results include: non-included variables could negatively affect equal comparisons between beats with and without ASAP, ASAP did not target purposely youth with crime history or history of victimization, the City of Houston often undertakes special police initiatives in areas with high crime so more suspects could have been identified and more victims reported, and beats without ASAP may have had other community programs, events, or functions that helped to prevent juvenile crime. There was an increase in 1999–2000 crime rate as compared to the 1998–1999 year. It may represent natural fluctuation. Perhaps, more extended after school programs over a long period of time may be needed to achieve desirable changes in crime prevention. No significant (p<.05) relationship existed between the number of ASAP programs in a beat and the crime variables, except for the variable of nonviolent victims. However more programs were set up in the beats with traditionally higher crime rates. Evaluators postulate that if ASAP programs were randomly assigned geographically, the relationships would have been different.
Workforce Development	Interviews with parents revealed the theme that ASAP helps working parents to meet their diverse responsibilities.
Youth Development	Facilitators who responded to the Facilitator survey, on average, considered that students made great gains in cognitive, affective, and social domains as a result of program participation. The mean scores on all survey items were significantly (p<.05) more positive than neutral.

A Profile of the Evaluation of After School Achievement Program

Profile Information