Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of After School Achievement Program

Program Description

Overview	The After School Achievement Program (ASAP) was founded in 1997 by the City of Houston, with the support of the Joint City/County Commission of Children and other community leaders, to provide after school programming to Houston's children and youth. The ASAP program has six goals: (1) to reduce crime committed by and against juveniles, (2) to prevent delinquency, (3) to provide a safe, supervised place for youth, (4) to provide academic enhancement and enrichment, (5) to promote school attendance and discourage school drop out, and (6) to motivate youth to develop good citizenship. The program has expanded from 7 sites in 1997 to 95 sites in 2000–2001.
Start Date	1997
Scope	local
Type	after school
Location	urban

Setting	public schools, community-based organizations, faith-based organizations
Participants	elementary through middle school students
Number of Sites/Grantees	95 sites, each receiving a grant in the range of $12,650 to $40,000
Number Served	11,649 children/month (2000–2001)
Components	All ASAP grantees must provide four major program components to students: 1. Academic enrichment (e.g., homework assistance, academic remediation, career awareness, and technology-based activities) 2. Personal skills development (e.g., leadership/good citizenship, team building, conflict management/violence prevention, and volunteerism) 3. Enrichment activities (e.g., recreation/sports, field trips, fine arts, and creative activities) 4. Community involvement (e.g., senior home visitations, community restoration projects, community/parental involvement, continuous feedback from teachers/parents, and adult/youth volunteerism)
Funding Level	$2.3 million (2000–2001). All grantees were required to provide a “dollar for dollar” match for additional operational expenses.
Funding Sources	City of Houston

Evaluation

Overview	Evaluations of the ASAP program were conducted annually for four years, from 1998 through 2001. Smith et al., of the University of Houston's College of Education, carried out all evaluations, focusing on ASAP attendance, program availability and variety, crime and academic impacts, and perceptions of quality by participants, parents, and Facilitators/teachers.
Evaluators	Dennis W. Smith et al., University of Houston, College of Education
Evaluations Profiled	After School Achievement Program (ASAP), 1998 Great Cities Do Great Things: Year 2 Evaluation of the After School Achievement Program (ASAP), 1999 Terrific Children: A City-Wide Program to Support Communities Year 3 Evaluation of the After School Achievement Program (ASAP), 2000 Shaping our Children's Future: Keeping a Promise in Houston Communities Year 4 Evaluation of the Mayor's After-School Achievement Program (ASAP), 2001
Evaluations Planned	not available
Report Availability	Smith, D. W., Zhang, J. J., Cheng, E. Y., & Lam, E. T. C. (1998). After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., Zhang, J. J., Rodriquez, A., Haynes, M., & Brimer, J. (1999). Great cities do great things: Year 2 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., Zhang, J. J., Brimer, J., & Rodriquez, A. (2000). Terrific children: A city-wide program to support communities Year 3 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston. Smith, D. W., & Zhang, J. J. (2001). Shaping our children's future: Keeping a promise in Houston communities Year 4 evaluation of the Mayor's After School Achievement Program (ASAP). Houston, TX: University of Houston.

Contacts

Evaluation	Dennis W. Smith Email: dwsmith@uh.edu
Program	Jennifer Brimer Sr. Community Liaison Coordinator After School Achievement Program City of Houston P.O. Box 1562 Houston, TX Tel: 713-437-6981 Email: jenniferbrimer@cityofhouston.net
Profile Updated	January 27, 2003

Evaluation 4: Shaping our Children's Future: Keeping a Promise in Houston Communities Year 4 Evaluation of the Mayor's After School Achievement Program (ASAP), 2001

Evaluation Description

Evaluation Purpose	To monitor the function and quality of the 2000–2001 City of Houston-funded ASAP program, particularly related to program impact in the areas of academic achievement and violence prevention. As this was the fourth year of study, evaluators looked for trends over time.
Evaluation Design	Non-Experimental and Quasi-Experimental: Data were collected from 95 sites and about participating children, with 72 of those sites providing survey data. Parent survey respondents were from 40 randomly selected schools. To study the impact of ASAP on academic achievement, 160 students from four school sites were recruited to be in treatment and comparison groups: 69 were participating in ASAP while 91 were not. The four schools included in this quasi-experimental component of the evaluation were similar in terms of sociodemographics. Most of the 160 students were Hispanic, from families with low incomes, and on free/reduced-price lunch.
Data Collection Methods	Document Review: Documents reviewed for the evaluation included ASAP records of program attendance and site curricular activities, school reports of attendance and grades, and archived police documents regarding crime data for local precincts. All 95 participating sites provided the ASAP and school documents as well as crime data from associated local police beats. Data were collected from September 2000 through June 2001. Interviews/Focus Groups: Semi-structured interviews were conducted with 39 ASAP facilitators/teachers, 48 parents, and 78 youth from 40 randomly selected sites about their perceptions of the value of the ASAP. Interviews were conducted in Spring 2001. Secondary Source/Data Review: Evaluations of ASAP for the last three years were analyzed for comparative purposes. Surveys/Questionnaires: Students, parents, and teachers were surveyed about their perceptions and opinions of program activities at their sites. All surveys were validated during Year 1 of the ASAP evaluation, including tests of content and construct, revised for use in the 1998–1999 evaluation, and tested and validated again in 1999–2000. Four-thousand-nine-hundred-thirty-seven children from 72 sites responded to the Youth Survey which consisted of 30 items in Likert scale format and focused on their perceptions of the program and their intention to participate next year. Two-thousand-one-hundred-thirty-five parents from 72 sites were surveyed about their perceptions of the quality of the ASAP program, their intention to send their children to ASAP in the future, and how their children's after school needs would be met in the absence of ASAP with a 33-item Likert scale format instrument. Both parent and student surveys were available in Spanish and English. Seven-hundred-eighty-nine Facilitators (teachers) from 72 sites responded to an 18-item Likert scale survey about their students' improvements in the cognitive, affective, and social domains as a result of participation in ASAP. The 1st ASAP Partnership Fair was evaluated by 107 participants using a 21-item Likert scale survey that looked at quality, workshop instruction, vendor exhibits, and exemplary programs. Tests/Assessments: The ASAP office coordinator visited each of the 95 sites and conducted a 22-item Likert 5-scale assessment adapted from the National Institute for Out-of-School Time and the 1998 Department of Education's booklet, Safe and Smart: Making After-School Hours Work for Kids. The assessment included items that fell into four categories: staff, programming and curricula, facilities and supplies, and administration. The four categories of the assessment had high alpha reliability coefficients indicating that they had sound internal consistency, supporting the credibility of their use for evaluation purposes. One-hundred-sixty students, a treatment and comparison group, from four schools were tested in 11 academic subject areas including reading, other language arts, mathematics, science, social studies, fine arts, handwriting, physical education, health and safety, computers, and science lab. The students were given a pretest at the beginning of the 2000–2001 school year and a posttest at the end of the school year. Posttests were different from pretests and tended to be of greater difficulty.
Data Collection Timeframe	Data were collected from September 2000 through June 2001.

Findings:
Formative/Process Findings

Activity Implementation	Programs generally followed program guidelines regarding the implementation of the four required programming areas: academic enhancement, personal skills, enrichment activity, and community involvement. Activities in these four areas comprised 95% of all program activities and were significantly (p<.05) more frequent than other activities. Academic activities accounted for 40% of programming, arts were 24%, sports 22%, and society 10%. Other less frequent categories of activities include livelihood, amusement, health, and other. Interviews with children revealed a theme that children feel that they participate in quality enrichment activities at ASAP.
Costs/Revenues	The annual cost per child was $200, down 10% from 1999–2000.
Parent/Community Involvement	Evaluators identified key factors critical to successful implementation of an after school program—“best practices”—one of which is regular parental involvement.
Program Context/Infrastructure	Programs operated a mean of 14.82 days per month. Of a total of 117 police beats, 50 had one or more ASAP programs. In each of these 50, there were 1–5 ASAP programs. ASAP police beats typically had higher crime rates than non-ASAP beats. ASAP typically enrolls only about one-third of the youth residing in a particular police beat. The City of Houston has increased the number of officers assigned to police beats with higher crime rates, the same beats more likely to have ASAP programs. Student survey respondents answered that if there were no after school program, 57% of them would be at home, 18% would be with friends, 9% would be with a relative, 7% would at some “other” location, 5% would be on the street, and 4% would be in the park. As to what they would be doing, 41% indicated that they would be doing homework, 20% would be watching TV, 15% would be playing, 14% would be hanging out, 5% would be doing nothing, and 4% would be doing some other activity. In the absence of an after school program, many would be supervised by parents (42%), some by no one (17%), some by a sibling (14%), some by another relative (13%), a few by friends (8%), and a small number would be supervised by some other entity (6%). Parents reported that if ASAP did not exist, their children would be taken care of by a relative, private sitter, or another organization (45%), by no one (22%), by a sibling (16%), or another arrangement (16%). Nonprofit organizations hosting ASAP programs offered more days and hours of programming than Houston Independent School District (HISD) sites, who, in turn, offered more than other school districts hosting ASAP sites. HISD ASAP sites had higher numbers of participants in each activity than other school district sties, who, in turn, had more participants per activity than nonprofit sites. All three types of sites—HISD, other school districts, and nonprofit organizations—were equally likely to serve African-American, Asian/Pacific-Islander, and American-Indian youth (p<.05). In contrast, for Hispanic youth, HISD sites were more common, followed by other public schools, and then by nonprofits. For Anglo-Caucasian youth, nonprofits were still the least frequent, but other public schools served more Anglo youth than HISD sites. There was no significant (p<.05) difference among the three types of sites in the monthly withdraws from the program. HISD sites had significantly (p<.05) more children enrolled each month, more new enrollments each month, and higher average daily attendance than other school district sites, which in turn were higher than nonprofit organization sites. No significant (p<.05) differences were found between the three types of sites in terms of the proportion of youth coming from moderate-income families. However, HISD sites were greater in the number of participants who were from low-income families, who were from female-headed families, and who received free/reduced-price lunch than other public school sites, and in turn greater than nonprofit sites. Program quality, according to the independent assessment conducted by the ASAP office coordinator, was similar for all three types of sites. Significant (p<.05) differences were found in the category of administration, but later analyses revealed no substantive difference.
Recruitment/Participation	Almost 12,000 children were served by ASAP on a monthly basis, an average of approximately 120 children per site, during the 2000–2001 year. More than 55% of parents had one child in ASAP while close to 40% of parents had two or three children in ASAP. Fifty-three percent of participating students were Hispanic, 41% were African-American, 4% were Anglo, and 2% were Asian. There were significantly (p<.05) more Hispanic and African-American students than other ethnicities. Eighty-eight percent of ASAP participants were between the ages of five and twelve and enrolled in the first through eighth grades. There were equal numbers of male and female children in ASAP. Forty percent of program participants came from homes headed by single mothers. About 35% of parents considered that their incomes were “inadequate” or “uncomfortable.” Seventy-three percent of program participants were from low-income families and 77% received free or reduced-price lunch (83% qualified). Ten percent came from families of moderate income. Less than 1% of participants were disabled. There were an average of 54.5 participants in each activity offered. Average daily attendance rates were approximately 64%. Average monthly enrollment was at 97% of total capacity and withdrawal rates were only about 3%. Of the total monthly participants, 30% were new enrollments. Monthly new enrollment was positively (p<.05) related to the number of participants from low- and moderate-income families, female-headed families, children receiving free/reduced-price lunch, and children from African-American and Hispanic ethnic backgrounds. The number of withdrawn participants was positively (p<.05) related to the number of participants from low-income families, female-headed households, participants receiving free/reduced-price lunch, and Hispanic youth.
Satisfaction	Seventy-six percent of surveyed participants reported “definitely” liking ASAP, 20% answered “kind of,” and 4% answered “not at all” to the same question. Significantly (p<.05) more participants felt positively about the program than did not. Sixty-nine percent of surveyed participants reported that they would “definitely” wanted to attend ASAP next year, 19% reported that they “kind of” wanted to, and 13% reported that they did “not at all” want to. Significantly (p<.05) more participants wanted to return to the program than did not. Student satisfaction survey responses revealed that participants were predominantly satisfied with ASAP. All mean scores were significantly (p<.05) more positive than neutral. Parent survey respondents indicated a high level of satisfaction with ASAP, usually answering “4” on a Likert 5-scale survey. Parents' overall rating of the program, as well as mean scores for all items, were significantly (p<.05) more positive than neutral. More than 90% of parents reported that they would recommend the program to other parents. Parents' positive perceptions of the functioning and offerings of ASAP were positively and significantly (p<.05) related to the number of their children in the program. Satisfaction and need were not significantly (p<.05) related to the number of children parents had in the program. Parents of younger children were significantly (p<.05) more satisfied with the program than parents of older children. Parents with lower family incomes were significantly (p<.05) more satisfied with the program than parents with higher family incomes. Parents of African-American and Hispanic youth were generally more satisfied with the quality of the program. Parents of Asian children provided average assessments of ASAP. Parents of American-Indian children were very satisfied with the overall program, but had reservations about program operation. Parents of Caucasian children were least satisfied with ASAP, of all ethnic groups. Parents whose children would be with a private sitter or a relative in the absence of ASAP were more satisfied overall with the program than those who had other arrangements. In other words, ASAP was more satisfactory to those who had the greatest child care needs. Participants were generally satisfied with the ASAP Partnership Fair, assessed with survey items falling into each of four categories: general quality, workshop instruction, vendor exhibits, and exemplary programs. Survey respondents predominantly chose “4” and “5” on a Likert 5-scale survey, with all mean scores significantly (p<.05) more positive than neutral. Parent interviews revealed a theme that parents feel ASAP must be a priority for the City of Houston.
Staffing/Training	Evaluators identified key factors critical to successful implementation of an after school program—best practices—one of which is dedicated teacher involvement. In interviews, facilitators/teachers expressed concern over low salaries, about $20/hour for staff and $7–$12/hour for teachers' aids. School district policies determine compensation. Social workers and nurses are not eligible to be paid staff members. Also, staff mentioned difficulty recruiting and maintaining volunteers.
Systemic Infrastructure	According to ASAP teacher surveys, the ASAP office was generally considered to be effective, with mean scores significantly (p<.05) above neutral. Evaluators identified key factors critical to successful implementation of an after school program—best practices—one of which is strong program management and accurate reporting.

Summative/Outcome Findings

Academic	Students attending ASAP scored significantly (p<.05) higher on the academic posttest in the areas of science and fine arts than student test takers in the comparison group. There were no significant (p<.05) differences in any of the other nine areas tested (reading, other language arts, mathematics, social studies, handwriting, physical education, health and safety, computers, and science lab), but posttest mean scores for program participants were all clearly higher than for comparison group members. According to interviews with students, a theme emerged that students perceive that ASAP is contributing to their improved academic performance.
Family	In interviews with parents, one theme that emerged was that parents depend on quality after school programs.
Prevention	All mean juvenile crime variables for the period between August 2000 and May 2001, between 3pm and 6pm, were significantly (p<.05) lower than the three previous years, but analyses of covariance indicated that differences between those beats with ASAP programs and those without were not statistically significant (p<.05).
Youth Development	Facilitators/teachers from 75 sites evaluated students using a Likert 5-scale survey. Predominantly, the teachers selected “4” and “5,” meaning that the teachers considered that as a result of participation in ASAP students had made great improvements in the cognitive, affective, and behavioral domains. The mean scores were all significantly (p<.05) greater than neutral (3). Facilitator interviews revealed a theme that facilitators feel that time after school with students creates trust among students, facilitators, and parents. Many facilitators commented that increased time after school with students allows them to build stronger relationships with both children and parents.

A Profile of the Evaluation of After School Achievement Program

Profile Information