You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼


Program Description

Overview The After School Achievement Program (ASAP) was founded in 1997 by the City of Houston, with the support of the Joint City/County Commission of Children and other community leaders, to provide after school programming to Houston's children and youth. The ASAP program has six goals: (1) to reduce crime committed by and against juveniles, (2) to prevent delinquency, (3) to provide a safe, supervised place for youth, (4) to provide academic enhancement and enrichment, (5) to promote school attendance and discourage school drop out, and (6) to motivate youth to develop good citizenship. The program has expanded from 7 sites in 1997 to 95 sites in 2000–2001.
Start Date 1997
Scope local
Type after school
Location urban
Setting public schools, community-based organizations, faith-based organizations
Participants elementary through middle school students
Number of Sites/Grantees 95 sites, each receiving a grant in the range of $12,650 to $40,000
Number Served 11,649 children/month (2000–2001)
Components All ASAP grantees must provide four major program components to students:

1. Academic enrichment (e.g., homework assistance, academic remediation, career awareness, and technology-based activities)
2. Personal skills development (e.g., leadership/good citizenship, team building, conflict management/violence prevention, and volunteerism)
3. Enrichment activities (e.g., recreation/sports, field trips, fine arts, and creative activities)
4. Community involvement (e.g., senior home visitations, community restoration projects, community/parental involvement, continuous feedback from teachers/parents, and adult/youth volunteerism)
Funding Level $2.3 million (2000–2001). All grantees were required to provide a “dollar for dollar” match for additional operational expenses.
Funding Sources City of Houston


Evaluation

Overview Evaluations of the ASAP program were conducted annually for four years, from 1998 through 2001. Smith et al., of the University of Houston's College of Education, carried out all evaluations, focusing on ASAP attendance, program availability and variety, crime and academic impacts, and perceptions of quality by participants, parents, and Facilitators/teachers.
Evaluators Dennis W. Smith et al., University of Houston, College of Education
Evaluations Profiled After School Achievement Program (ASAP), 1998

Great Cities Do Great Things: Year 2 Evaluation of the After School Achievement Program (ASAP), 1999

Terrific Children: A City-Wide Program to Support Communities Year 3 Evaluation of the After School Achievement Program (ASAP), 2000

Shaping our Children's Future: Keeping a Promise in Houston Communities Year 4 Evaluation of the Mayor's After-School Achievement Program (ASAP), 2001
Evaluations Planned not available
Report Availability Smith, D. W., Zhang, J. J., Cheng, E. Y., & Lam, E. T. C. (1998). After School Achievement Program (ASAP). Houston, TX: University of Houston.

Smith, D. W., Zhang, J. J., Rodriquez, A., Haynes, M., & Brimer, J. (1999). Great cities do great things: Year 2 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston.

Smith, D. W., Zhang, J. J., Brimer, J., & Rodriquez, A. (2000). Terrific children: A city-wide program to support communities Year 3 evaluation of the After School Achievement Program (ASAP). Houston, TX: University of Houston.

Smith, D. W., & Zhang, J. J. (2001). Shaping our children's future: Keeping a promise in Houston communities Year 4 evaluation of the Mayor's After School Achievement Program (ASAP). Houston, TX: University of Houston.


Contacts

Evaluation Dennis W. Smith
Email: dwsmith@uh.edu
Program Jennifer Brimer
Sr. Community Liaison Coordinator
After School Achievement Program
City of Houston
P.O. Box 1562
Houston, TX
Tel: 713-437-6981
Email: jenniferbrimer@cityofhouston.net
Profile Updated January 27, 2003

Evaluation 1: After School Achievement Program (ASAP), 1998



Evaluation Description

Evaluation Purpose To describe the first year of ASAP program operation, from January 1998 to May 1998, with respect to the following five areas: curriculum formulation, activity programs, teacher evaluations of students, parents' opinions, and student perceptions.
Evaluation Design Non-Experimental: Data were gathered from eight elementary schools and one middle school (nine of the eleven original sites). All samples were convenience samples.
Data Collection Methods Surveys/Questionnaires: Program coordinators from each school, of whom there were nine total, completed two forms: one which had them answer open-ended questions about the general program description (including program name, enrollment, general program description, duration, and specific activities to enhance academics, personal skills, enrichment options, and community involvement) and another which included open-ended questions about an average weekly lesson plan (only eight of nine completed the lesson plan).

Teachers (n=380) from six of the nine schools responded to a Likert-5 scale formatted survey about perceived program outcomes in the cognitive, affective, and social domains on their participating students.

Parents from eight of the nine schools (n=280) responded to surveys covering perceived program quality, intention to continue to send children, and suggestions.

Students (n=533) from all nine schools school responded to surveys about the pros and cons of the program.

Parent and student surveys contained both Likert-5 scale questions and open-ended questions.
Data Collection Timeframe All data were collected between March and June of 1998.


Findings:
Formative/Process Findings

Activity Implementation A total of 65 activity programs were provided by the schools relating to areas, such as academics, professional development, life skills, arts and crafts, sport/exercise, domestics, games, field trips, and social activity.

Among the most adopted programs were arts and crafts, computer class, homework/tutorial, mathematics, reading, swimming, and basketball.

Each program had one or more of the following incorporated in its content/lesson plans and class implementations: academic enhancement, personal skills, enrichment opportunities, and community involvement.
Parent/Community Involvement Of the 22% of parents responding to the question, significantly (p<.001) more parents than not, 51 versus 11, answered that they would be willing to help out with the program the following year.
Program Context/Infrastructure When asked what they would be doing if there were no after school program, 84% of students who answered this question were significantly more likely to give one of the following answers than to give other answers: stay at home, do homework, watch TV, play, hang out with friends/play outside, sleep, or do nothing.
Satisfaction Ninety-four percent of students indicated that they liked the program. There were significantly (p<.000) more students who liked the program than who did not.

Ninety percent of students indicated that they would like to attend the program next year. There were significantly (p<.000) more students who indicated that they would like to return than who did not.

When asked about what activity they liked best in the program, the 92% of students who responded to the question gave specific answers that fell into the following five categories: activity, learning, enjoyment, people, and miscellaneous.

The five most liked activities by students were: arts and crafts, dance, computer, soccer, and basketball. These five occurred significantly (p<.05) more frequently than other favorite activities.

When asked about what activity they liked least in the program, the 82% of students who responded to the question gave specific answers that fell into the following five categories: disliked activity, disliked management, disliked work, disliked in general, and disliked nothing. Eighty students disliked nothing.

The five least-liked activities by students were: arts and crafts, homework, dance, piano, and pet care. These five occurred significantly (p<.05) more frequently than other least-liked activities.

When asked what activities they would like to do next year, the 92% of students who responded to that item gave specific answers in the following eight categories: arts and crafts, physical activity, academics, domestics, game/video, field trip, social group, and miscellaneous.

Of the artistic activities students said they would like to do next year, significantly (p<.05) more students said dance, arts and crafts, and piano as compared to other artistic activities.

Of the physical activities students said they would like to do next year, significantly (p<.05) more students mentioned basketball, cheerleading, swimming, and soccer as compared to other physical activities.

Of the academic activities students said that they would like to do next year, significantly (p<.05) more students said computer/Internet as compared to other academic activities.

When asked what they would change about the program, the 89% of students who responded to this item gave specific answers in the following six categories: program, food/beverage, schedule, facility/equipment, staff, and management. The most frequent responses, significantly (p<.05) more frequent than other responses, were a desire for more and better snacks and longer program time. More than 100 students said that they liked the current programs and would change nothing.

Parents were strongly supportive of the programs. In other words, the mean scores for all the Likert 5-scale satisfaction items were significantly (p<.05) greater than 3 (neutral). And when asked to provide a 5-scale overall rating of the program, the mean response was 4.48.

Using a Likert 3-scale, over 90% of parents indicated that they would recommend the program to other parents.

When asked what they would change about the program, the 61% of parents who responded to this item gave specific answers in the following four categories: program, participation, schedule, and personnel.

When asked what the best thing about the after school program was, the 70% of parents who responded to this item gave specific answers in the following seven categories: cognitive/skill development, affective/behavioral development, extracurricular activity, safety and something to do, positive evaluation, operational excellence, and constructive suggestions.

When asked about what their children say about the after school program, the 71% of parents who responded to this item gave specific answers in the following three categories: positive evaluation, future intention, and complaints, of which there were only two. The response rate and comments suggest that children enjoyed the program and intended to participate in a similar program in the future.


Summative/Outcome Findings

Academic Teachers perceived students to have made substantial cognitive improvements as a result of participation in the after school programs. On the Likert 5-scale, the mean scores on the cognitive improvement items were significantly (p<.05) greater than 3 (neutral).
Youth Development Teachers perceived students to have made great progress in the affective and behavioral domains as a result of participation in the after school programs. In other words, the mean scores on the affective and behavioral improvement survey items were significantly (p<.05) greater than 3 (neutral).

Teachers perceived significant (p<.05) differences in impact for differently aged children. Teachers of sixth graders reported the most substantial improvements, compared to all other grades, in cognitive, affective, and behavioral domains. Kindergarten and first graders, according to their teachers, were reported to receive more benefits from the program than second and fifth graders, according to their teachers, respectively.

When asked what they had learned through program participation, the 83% of students who responded to this item gave specific answers in the following eight categories: general, affective, academics and skills, arts/crafts, sports, recreation, professional ability, and living skills. There were significantly (p<.05) greater frequencies for the following specific answers: a lot of good things, fun/cool stuff, reading, math, computer, English, dance, and arts/crafts.

Evaluation 2: Great Cities Do Great Things: Year 2 Evaluation of the After School Achievement Program (ASAP), 1999



Evaluation Description

Evaluation Purpose To examine the functioning and quality of the After School Achievement Program during the 1998–1999 school year, with a focus on impacts on academic achievement and violence prevention.
Evaluation Design Non-Experimental: Data collection occurred at 58 of the 68 existing program sites. Twenty sites were randomly selected for more intensive study of student, teacher, and parent perceptions. Of these, 17 provided usable student data, 17 provided usable parent data, and all 20 provided usable teacher data. The students that the teachers evaluated were selected by the teachers.
Data Collection Methods Document Review: Documents reviewed for the evaluation included ASAP records of program attendance, participants, budget, and site curricular activities, school reports of attendance and grades, and archived police documents regarding crime data for local precincts. The ASAP records and school reports were collected for all 58 participating sites. The police reports were collected for only 20 sites

Surveys/Questionnaires: Students, parents, and teachers were surveyed about their perceptions and opinions of program activities at their sites. All surveys were validated during Year 1 of the ASAP evaluation, including tests of content and construct, and revised for use in the 1998–1999 evaluation. One-thousand-twenty-three students at 17 program sites were surveyed about their perceptions of the program and their intentions for future participation using a 27-item Likert 3-scale survey for all but two questions where satisfied, dissatisfied, and neutral were represented by smiling, neutral, or frowning faces. Five-hundred-two parents from 17 sites were surveyed about their perceptions of the quality of the program, their intentions to send their child in the future, and their suggestions using a 32-item Likert 5- and 3-scale survey. Teachers from 20 sites were surveyed about 182 student ASAP participants to determine teachers' perceptions of individual student improvements in the cognitive, affective, and social domains as a result of ASAP participation. The teachers selected which students to evaluate. The teacher surveys were comprised of 18-items in Likert 5-scale format.
Data Collection Timeframe Data collection occurred between September 1998 and June 1999.


Findings:
Formative/Process Findings

Activity Implementation In general, ASAP programs followed the City of Houston program guidelines with respect to activities and offered activities that cut across the four component areas of academic enhancement, personal skills, enrichment activity, and community involvement.

Two-hundred-fifty different activities were cumulatively offered by sites in the areas of academics, professional development, life skills, arts and crafts, sport/exercise, domestics, games, field trips, and social activity.

The most commonly adopted activities include: arts and crafts, computer instruction, homework/tutorial, Texas Assessment of Academic Skills (TAAS) practice, team building, leadership, recreation and sports, and community collaboration.

The 10 most frequent “academic enhancement” activities, significantly (p<.05) more common than other activities, in order, are: homework assistance, computer classes, TAAS, reading, math workshop, academic remediation, career awareness, technology activities, science, and group story time.

The 10 most frequent “personal skills” activities, significantly (p<.05) more common than other activities, in order, are: team building games, good citizenship, leadership, conflict management/violence prevention, volunteerism, cooperative learning, following directions, accident prevention, character education, and drug free school club.

The 10 most frequent “enrichment” activities, significantly (p<.05) more common than other activities, in order (from most to least frequent), are: recreation and sports, arts and crafts, dance, basketball, field trip, soccer, gymnastics music, fine arts, and aerobics.

The 10 most frequent “community involvement” activities, significantly (p<.05) more common than other activities, in order (from most to least frequent), are: volunteer, parents volunteer, collaboration (teachers, parents, school), continuous feedback, community/parent contribution, read write now, adult computer class, multicultural craft, parent conferences, and teacher aid volunteer.
Costs/Revenues Ninety-five percent of programs operate within the approved budget.

The basic program cost was $200 per participant in addition to some supplementary money from the Child Safety fund.
Program Context/Infrastructure Over 75% of program sites adopted a five-day per week schedule and most had operating hours of 2:30pm to 6:30pm.

Approximately 90% of programs served a snack. The YMCA of Greater Houston or Chapter I/Title I were primary providers of snacks.

All program sites followed program reporting requirements and approximately two thirds indicated that their program reports were complete and accurate.

Fifty-nine percent of student survey respondents indicated that if there were no after school program they would be at home; 13% said they would be with a friend; 9% said they would be with a relative; 7% said that they would be on the street; 7% said they would be at the park; 3% answered “other.” These findings were statistically significant (p<.000). There were no findings about the level of supervision at any of these sites.

Student survey respondents indicated that if there were no after school program they would be engaged in the following activities with the correspondent frequencies: 28% would be watching TV, 26% would be doing homework, 23% would be playing, 12% would be hanging out, 7 % would be doing nothing, and 3% would be doing some other activity. These findings were statistically significant (p<.000). There were no findings about the level of supervision during these activities.

1998–1999 ASAP sites were located in police precincts with elevated crime rates-more than 55% of crime victims and suspects according to the 1997–1998 data were in ASAP located police precincts.
Recruitment/Participation Based on responses from 58 of 68 total sites, there were 7,700 participants in ASAP during the 1998–1999 school year.

Participants were 51% Hispanic and 41% African-American with these ethnicities significantly (p<.000) more common than others. Approximately 5% were Asian/Pacific-Islander, 3% were Anglo/Caucasian, .01% were American-Indian, and .01% were “other.”

Seventy-eight percent of participants receive free or reduced-price lunch. Forty-one percent of participants were from homes run by single mothers.

Sixty percent of participants attended the program on a daily basis.

A majority of program sites began operation between September and November of 1998 and ran until May 1999.

Most of the programs had enrollment capacities of either 50–99 children (47%) or 100–199 (33%). Three-quarters of sites were able to reach a minimum of 60 participants. Of the 25% who were not, a substantial proportion cited reporting difficulties rather than barriers to participation.
Satisfaction Ninety percent of all student survey respondents, who comprised 46% of all students participating in the program at the surveyed schools, indicated that they liked the ASAP program, 6% were unsure, and 3% indicated that they did not. There were significantly (p<.000) more students who felt positively about the program than students who did not.

Seventy-eight percent of all student survey respondents indicated that they wanted to attend the program next year, 14% were unsure, and 9% indicated that they did not want to. There were significantly (p<.000) more students who wanted to attend the program next year than students who did not.

The majority (ranging from 78% to 92%) of student survey respondents indicated that they were satisfied regarding the remaining 25 survey items. Statistical tests revealed that there were significantly (p<.05) more students who felt satisfied than students who did not.

Of the 520 parents from 17 randomly selected schools who responded to the parent survey, 50% of the eligible parents, 77% gave the ASAP program an overall rating of “excellent.” In addition, parents predominantly gave the most positive response on the remaining Likert 4-scale survey items, indicating widespread satisfaction with the program. Mean scores on all parent survey items were found to be significantly (p<.05) greater than neutral (2.5).

Ninety-eight percent of parents indicated on the parent survey that they intended to enroll their child in the ASAP program the following year. This finding was statistically significant (p<.05).

Parents' positive perceptions of the program fell across all five content areas including general operation, leadership quality, program materials, future intentions, and personal needs.

The evaluators gave the ASAP program an “A” in the domain of “student evaluation” since the program consistently yielded high student ratings.

The evaluators gave the ASAP program an “A” in the domain of “parent evaluation” since the parents were extremely positive about the ASAP program.
Staffing/Training Each program site had an average of 11 paid staff members and 6 volunteer staff members. The paid staff averaged less than one hour/week/person while the volunteer staff averaged more than four hours/week/person.


Summative/Outcome Findings

Academic Each program site had over 80 students, usually more than 60% of all participants, who improved their academic grades, school attendance, and school behaviors. The percentage of students who improved their grades on average was significantly (p <.05) greater than 50%.
Prevention 1997–1998 and 1998–1999 data from the Houston police precincts showed a dramatic reduction in youth victimization rates in precincts with ASAP programs compared with a much smaller reduction in non-ASAP precincts. During 1998–1999, the decline in serious crimes involving a juvenile victim was 15% in ASAP districts and 12% for all crimes involving a juvenile victim. The decline is more than three times that of non-ASAP precincts for serious crimes against juveniles and more than 1.5 times that of non-ASAP precincts for all youth victimizations. The difference between ASAP and non-ASAP precinct crime reductions was statistically significant (p<.05) both for the total of all violent crimes where juveniles were victims and the total of all crimes in which juveniles were victims.

1997–1998 and 1998–1999 data from Houston police precincts showed a marked decline in crime where youths were suspects in the districts with ASAP programs compared with a much smaller reduction in non-ASAP districts. During 1998–1999, ASAP precincts experienced a 13% decline in serious crimes involving a youth suspect and total crimes involving a youth suspect. The ASAP precinct decline was more than 15 times that of non-ASAP precincts for serious crimes with a youth suspect and more than 1.2 times the non-ASAP reduction in total youth crime suspects. The proportional reduction in violent crimes with youth suspects and all crimes with youth suspects between ASAP and non-ASAP precincts was statistically significant (p<.05).

For every $2 per month spent on the children in the program, a one percent decline in violent crimes and suspects was realized in police precincts with ASAP sites compared to precincts without ASAP sites.
Youth Development Teachers evaluating 182 students from 20 randomly selected sites, approximately 10% of students in the ASAP programs at their schools, predominantly indicated that students had made progress in the cognitive, affective, and behavioral areas as a result of participation in the ASAP program. Consistently the mean scores on the 18 survey items in Likert 5-scale format were significantly (p<.05) greater than neutral (3).

The highest mean items on the teacher evaluations of students, both having a mean of 4.6/5, were related to children having adults who care about them in the after school program.

Evaluation 3: Terrific Children: A City-Wide Program to Support Communities Year 3 Evaluation of the After School Achievement Program (ASAP), 2000



Evaluation Description

Evaluation Purpose To explore the function and quality of the 1999–2000 ASAP program, with a focus ASAP's impact on academic achievement and violence prevention. Additionally, the evaluation looked at trends established over the three-year period of the program's operation.
Evaluation Design Non-Experimental: Data were collected from 111 sites and about 11,786 children. Forty sites, and 3066 children and 1,695 parents at those sites, were randomly selected for more intensive study of parent and student perceptions. Crime data were gathered from 103 ASAP sites and local beats.
Data Collection Methods Document Review: Documents reviewed for the evaluation included ASAP records of program attendance and site curricular activities, school reports of attendance and grades, and archived police documents regarding crime data for local precincts. All 111 participating sites provided the ASAP and school documents, but only 103 provided the police documents. Data was collected from September 1999 through June 2000.

Interviews/Focus Groups: Semi-structured interviews were conducted with 10 facilitators, 46 parents, and 55 children about their perceptions of the value of the ASAP. Interviews were conducted in July and August 2000.

Surveys/Questionnaires: Students, parents, and teachers were surveyed about their perceptions and opinions of program activities at their sites. All surveys were validated during Year 1 of the ASAP evaluation, including tests of content and construct, revised for use in the 1998–1999 evaluation, and tested and validated again in 1999–2000.

Three-thousand-sixty-six children from 40 sites responded to the Youth Survey, which consisted of 30 items in Likert scale format and focused on their perceptions of the program and their intention to participate next year.

One-thousand-six-hundred-ninety-five parents from 40 sites were surveyed about their perceptions of the quality of the ASAP program, their intention to send their children to ASAP in the future, and how their children's after school needs would be met in the absence of ASAP with a 33-item Likert scale format instrument. Both parent and student surveys were available in Spanish and English.

Five-hundred-three Facilitators (teachers) from 103 sites responded to an 18-item Likert scale survey about their students' improvements in the cognitive, affective, and social domains as a result of participation in ASAP.
Data Collection Timeframe Data were collected from September 1999 through June 2000.


Findings:
Formative/Process Findings

Activity Implementation On average, there were 61 children in each activity offering.

Over all three years of program operation, ASAP programs consistently followed the stated curriculum and its four objectives (academic enhancement, personal skills, enrichment activity, and community involvement).

Program activities of ASAP sites generally followed the program guidelines in that there were activities in all four domains: academic enhancement, personal skills, enrichment activity, and community involvement. Activities that fell into these four domains had significantly (p<.05) greater frequencies than other activities.

Interviews with children revealed the theme that they are engaged in personally rewarding activities.
Costs/Revenues The per child annual cost was about $225, 10% more than in 1998–1999, but still approximately equal in cost to one month of privately arranged care.

Variables used to define cost effectiveness include the projected per site funding by the City of Houston, the number of paid personnel, a constant personnel/operational expense projection, and the average number of children served.

Total cost was $1.63/day/child for ASAP. The operational (excluding personnel) costs of ASAP were $.41/day/child.

The funded personnel costs were about $9.25/hour/worker and a cost per child of $1.22/child/day.

Volunteers and in-kind contributions from each site supplemented the ASAP budget.
Program Context/Infrastructure The evaluators sought to understand what ASAP participants would be doing in the absence of the program.

Fifty-three percent of student survey respondents indicated that they would be at home if there were no after school program; 7.2%, or 221 children, responded that they would be on the street.

If there were no after school program, about half of student survey respondents indicated that they would be doing homework or playing; 25% indicated that they would be doing nothing or hanging out; and 20% indicated that they would be watching TV.

Approximately 50% of student survey respondents indicated that they would have adult supervision if there were no after school program, while 20% said that they would have no one to watch them.

Forty percent of parent survey respondents indicated that if there were no after school program, their children would be supervised by a relative or sitter; 30% indicated that their children would be by themselves; and 20% indicated that their children would be watched by a sibling.

Interviews with children revealed the theme that ASAP participants would be home alone without the program.

Programs operated for an average of 16 days per month and 3 hours per day.

Significant (p<.05) differences existed in terms of program operation variables among different types of ASAP program sites.

Nonprofit organizations that hosted ASAP sites offered more days and hours of operation than Houston Independent School District (HISD) sites, which in turn, offered more days and hours monthly than other public school sites. However, nonprofit and HISD sites offered comparable ASAP hours per day, which were significantly (p<.05) greater than other public school site hours/day.

No significant (p<.05) differences were found in academic outcomes between the three types of program sites (HISD, other public schools, and nonprofit organizations).

HISD sites had significantly (p<.05) more participants per activity than other public schools who, in turn, had more participants per activity than nonprofit ASAP sites.

Generally speaking, the number of participants in different program activities were similar for the three different host organizations (nonprofits, HISD schools, and other public schools). However, arts/crafts activities had significantly (p<.05) more participants per at HISD sites than at nonprofits which had more participants in arts/crafts than other public school ASAP sites. For athletics/recreation activities, HISD and other public school ASAP sites had significantly (p<.05) more participants per activity than nonprofits did. And for tutorial/homework, other public school sites had significantly (p<.05) more children per session than HISD, which had more than nonprofit organizations hosting ASAP.

There was no significant difference between the enrollments of African-American and American-Indian youth served in each of the three program venue types. Significant (p<.05) differences did exist in terms of Asian, white, and Hispanic enrollment: more Hispanics were served in HISD school sites, more Asians were served in other public school sites, and more whites were served in HISD and other public school sites.

No significant (p<.05) differences were found between the three types of programs in terms of new enrollments or the number of youth from female-headed households.

Significant (p<.05) differences between the three types of sites were found in the total monthly enrollment, overall program withdrawal, youth receiving fee or reduced-price lunch, and average daily attendance. On all of these variables, HISD school sites were greater than other public school sites, which were, in turn, greater than nonprofit sites.
Recruitment/Participation Over three years of program participation, ASAP consistently served Hispanic and African-American youth, low-income youth, and youth from single parent families. Cumulatively, Hispanic youth represented 50% of all participants and African-American youth represented 41%; 80% of all participants received free or reduced-price lunch; and youth from female-headed single parent families comprise 40% of all participants.

The number of ASAP sites and participants increased each of its three years of operation from 11 sites and 1,500 participants in 1998, to 68 sites with 7,700 participants in 1998–1999, to 111 sites with over 12,000 children in 1999–2000.

Approximately 11,786 children were served in the ASAP program during the 1999–2000 school year each month based on reports from 111 of the 112 sites, a quarter of which were new enrollments.

The monthly average was approximately 100 children per site.

Seventy-eight percent of participants receive free or reduced-price lunch, while 88% qualify. Thirty-seven percent of participants are from homes with single mothers.

An average of 69% attended ASAP on a daily basis.

Forty-eight percent of ASAP participants were Hispanic; 45% of ASAP participants were African-American; 4.5% were Anglo; and 2.1% were Asian. Statistical tests reveal that the percentages of African Americans and Hispanics were significantly (p<.000) greater than the proportions of Asians/Pacific Islanders, Anglo/Caucasians, American Indians, and other ethnic groups.

There were approximately equal numbers of male and female children.

Fifty-five percent of parent survey respondents had one child in ASAP and 45% had two or three children in the program.

Ninety-seven percent of children were between the ages of 5 and 12 and in the first through eighth grades.

About 46% of parent survey respondents considered their incomes inadequate or uncomfortable.

Most of the programs met the requirement of having 60+ participants on a daily basis.

Sites with higher numbers of youth from female-headed households and/or are receiving free or reduced-price lunch had higher numbers of participants.

The number of youth receiving free or reduced-price lunch and the number of Hispanic youth were positively related to the number of new monthly enrollments, testifying to the value of such programs in Hispanic communities and the need for outreach into such communities to recruit and retain participants.

The number of youth receiving free or reduced-price lunch, the number of youth from female-headed households, the number of Hispanic youth, and the number of white youth were all positively associated with the number of youth withdrawing from the program each month, so evaluators conclude that program retention efforts should focus on youth with these backgrounds.
Satisfaction Over three years of program operation, there was no significant change in teacher perceptions of the program's impact on participating students or student and parents' satisfaction with the program. All were consistently positive.

Seventy-seven percent of student survey respondents indicated that they “definitely” liked ASAP, while 18% said they “kind of” liked ASAP, and the remaining 5% said that they did “not at all” like ASAP. Significantly (p<.000) more students felt positively about the program than negatively.

Seventy-one percent of student survey respondents indicated that they “definitely” wanted to attend ASAP next year, while 17% said they “kind of” wanted to attend ASAP next year, and the remaining 12% said that they wanted “not at all” to attend ASAP next year. Significantly (p<.000) more students wanted to attend ASAP next year than did not.

There were significantly (p<.05) more positive student survey responses than negative ones, indicating overall satisfaction with ASAP. In addition, on every survey item, a majority of students gave the most positive response.

Parent survey respondents gave high ratings to the ASAP program, with the majority giving the program the most positive rating on almost all of the survey items. All mean scores were significantly (p<.05) more positive than neutral.
Parents' perception of the functioning and offerings of the ASAP program were positively and significantly (p<.05) associated with the number of their children in ASAP, i.e., the more children of theirs in the program, the more positively they felt about the functioning and offerings of ASAP.

Parents' perceptions of program functioning were significantly (p<.05) negatively related to the age of their participating child. Parents of younger children were relatively more satisfied with the program than parents of older children.

Parents' perception of the program's leadership were significantly (p<.05) negatively related to family income which means that parents of low-income children were relatively more satisfied with the program than parents of higher income children.

Parents' satisfaction with program leadership and operation was found to be greater among parents of white and African-American children and lesser among parents of Asian-American and American-Indian children. American-Indian parents were satisfied with program leadership, but not program operation.

Parents who indicated that in the absence of ASAP they themselves, a relative, or a sitter would care for children were more satisfied with program functioning. Parents answering “a relative” or “themselves” were also more satisfied with program leadership. Parents answering “by themselves” or “by another organization” were more satisfied with program operation. These findings may suggest that ASAP was more satisfactory to those who had the most needs.

Parent evaluation of ASAP was found to be unrelated (p<.05) to children's gender or their qualification for free or reduced-price lunch.


Summative/Outcome Findings

Academic Data from the 1998–1999 and 1999–2000 evaluations revealed that ASAP programs consistently had at least 50% of program participants who made improvement in academic performance, school attendance, and school behaviors, respectively.

On average, each program had more than 50 students, or 50% of ASAP participants, who improved school grades, school attendance, and school behavior. The percentage to have improved in this way was found to be significantly (p<.05) greater than 25% and not significantly (p<.05) different from 50%.

Program enrollment size, number of new enrollments, number of withdraws, and daily attendance was not found to be significantly (p<.05) related to youth school grades, attendance, and behaviors, indicating that program size did not affect quality of the program.
Community Development Interviews with facilitators revealed the theme that ASAP enhances the quality of life in communities. Interviews with parents revealed the theme that ASAP creates support networks within the community.
Prevention Out of a total of 117 police beats, 57 had one or more ASAP programs, ranging from one to six.

There were no significant (p<.05) differences between beats with after school programs and those without for all crime variables, including number of violent suspects, number of nonviolent suspects, number of violent victims, and number of nonviolent victims. Considering that crime histories were different between the beats with and without the programs, juvenile crime data for the 1997–1998 and 1998–1999 years were used as covariates and their effects were partialled out in the analyses.

Possible reasons for non-significant crime results include: non-included variables could negatively affect equal comparisons between beats with and without ASAP, ASAP did not target purposely youth with crime history or history of victimization, the City of Houston often undertakes special police initiatives in areas with high crime so more suspects could have been identified and more victims reported, and beats without ASAP may have had other community programs, events, or functions that helped to prevent juvenile crime.

There was an increase in 1999–2000 crime rate as compared to the 1998–1999 year. It may represent natural fluctuation. Perhaps, more extended after school programs over a long period of time may be needed to achieve desirable changes in crime prevention.

No significant (p<.05) relationship existed between the number of ASAP programs in a beat and the crime variables, except for the variable of nonviolent victims. However more programs were set up in the beats with traditionally higher crime rates. Evaluators postulate that if ASAP programs were randomly assigned geographically, the relationships would have been different.
Workforce Development Interviews with parents revealed the theme that ASAP helps working parents to meet their diverse responsibilities.
Youth Development Facilitators who responded to the Facilitator survey, on average, considered that students made great gains in cognitive, affective, and social domains as a result of program participation. The mean scores on all survey items were significantly (p<.05) more positive than neutral.

Evaluation 4: Shaping our Children's Future: Keeping a Promise in Houston Communities Year 4 Evaluation of the Mayor's After School Achievement Program (ASAP), 2001



Evaluation Description

Evaluation Purpose To monitor the function and quality of the 2000–2001 City of Houston-funded ASAP program, particularly related to program impact in the areas of academic achievement and violence prevention. As this was the fourth year of study, evaluators looked for trends over time.
Evaluation Design Non-Experimental and Quasi-Experimental: Data were collected from 95 sites and about participating children, with 72 of those sites providing survey data. Parent survey respondents were from 40 randomly selected schools. To study the impact of ASAP on academic achievement, 160 students from four school sites were recruited to be in treatment and comparison groups: 69 were participating in ASAP while 91 were not. The four schools included in this quasi-experimental component of the evaluation were similar in terms of sociodemographics. Most of the 160 students were Hispanic, from families with low incomes, and on free/reduced-price lunch.
Data Collection Methods Document Review: Documents reviewed for the evaluation included ASAP records of program attendance and site curricular activities, school reports of attendance and grades, and archived police documents regarding crime data for local precincts. All 95 participating sites provided the ASAP and school documents as well as crime data from associated local police beats. Data were collected from September 2000 through June 2001.

Interviews/Focus Groups: Semi-structured interviews were conducted with 39 ASAP facilitators/teachers, 48 parents, and 78 youth from 40 randomly selected sites about their perceptions of the value of the ASAP. Interviews were conducted in Spring 2001.

Secondary Source/Data Review: Evaluations of ASAP for the last three years were analyzed for comparative purposes.

Surveys/Questionnaires: Students, parents, and teachers were surveyed about their perceptions and opinions of program activities at their sites. All surveys were validated during Year 1 of the ASAP evaluation, including tests of content and construct, revised for use in the 1998–1999 evaluation, and tested and validated again in 1999–2000.

Four-thousand-nine-hundred-thirty-seven children from 72 sites responded to the Youth Survey which consisted of 30 items in Likert scale format and focused on their perceptions of the program and their intention to participate next year.

Two-thousand-one-hundred-thirty-five parents from 72 sites were surveyed about their perceptions of the quality of the ASAP program, their intention to send their children to ASAP in the future, and how their children's after school needs would be met in the absence of ASAP with a 33-item Likert scale format instrument. Both parent and student surveys were available in Spanish and English.

Seven-hundred-eighty-nine Facilitators (teachers) from 72 sites responded to an 18-item Likert scale survey about their students' improvements in the cognitive, affective, and social domains as a result of participation in ASAP.

The 1st ASAP Partnership Fair was evaluated by 107 participants using a 21-item Likert scale survey that looked at quality, workshop instruction, vendor exhibits, and exemplary programs.

Tests/Assessments: The ASAP office coordinator visited each of the 95 sites and conducted a 22-item Likert 5-scale assessment adapted from the National Institute for Out-of-School Time and the 1998 Department of Education's booklet, Safe and Smart: Making After-School Hours Work for Kids. The assessment included items that fell into four categories: staff, programming and curricula, facilities and supplies, and administration. The four categories of the assessment had high alpha reliability coefficients indicating that they had sound internal consistency, supporting the credibility of their use for evaluation purposes.

One-hundred-sixty students, a treatment and comparison group, from four schools were tested in 11 academic subject areas including reading, other language arts, mathematics, science, social studies, fine arts, handwriting, physical education, health and safety, computers, and science lab. The students were given a pretest at the beginning of the 2000–2001 school year and a posttest at the end of the school year. Posttests were different from pretests and tended to be of greater difficulty.
Data Collection Timeframe Data were collected from September 2000 through June 2001.


Findings:
Formative/Process Findings

Activity Implementation Programs generally followed program guidelines regarding the implementation of the four required programming areas: academic enhancement, personal skills, enrichment activity, and community involvement. Activities in these four areas comprised 95% of all program activities and were significantly (p<.05) more frequent than other activities.

Academic activities accounted for 40% of programming, arts were 24%, sports 22%, and society 10%. Other less frequent categories of activities include livelihood, amusement, health, and other.

Interviews with children revealed a theme that children feel that they participate in quality enrichment activities at ASAP.
Costs/Revenues The annual cost per child was $200, down 10% from 1999–2000.
Parent/Community Involvement Evaluators identified key factors critical to successful implementation of an after school program—“best practices”—one of which is regular parental involvement.
Program Context/Infrastructure Programs operated a mean of 14.82 days per month.

Of a total of 117 police beats, 50 had one or more ASAP programs. In each of these 50, there were 1–5 ASAP programs.

ASAP police beats typically had higher crime rates than non-ASAP beats.

ASAP typically enrolls only about one-third of the youth residing in a particular police beat.

The City of Houston has increased the number of officers assigned to police beats with higher crime rates, the same beats more likely to have ASAP programs.

Student survey respondents answered that if there were no after school program, 57% of them would be at home, 18% would be with friends, 9% would be with a relative, 7% would at some “other” location, 5% would be on the street, and 4% would be in the park.

As to what they would be doing, 41% indicated that they would be doing homework, 20% would be watching TV, 15% would be playing, 14% would be hanging out, 5% would be doing nothing, and 4% would be doing some other activity.

In the absence of an after school program, many would be supervised by parents (42%), some by no one (17%), some by a sibling (14%), some by another relative (13%), a few by friends (8%), and a small number would be supervised by some other entity (6%).

Parents reported that if ASAP did not exist, their children would be taken care of by a relative, private sitter, or another organization (45%), by no one (22%), by a sibling (16%), or another arrangement (16%).

Nonprofit organizations hosting ASAP programs offered more days and hours of programming than Houston Independent School District (HISD) sites, who, in turn, offered more than other school districts hosting ASAP sites.

HISD ASAP sites had higher numbers of participants in each activity than other school district sties, who, in turn, had more participants per activity than nonprofit sites.

All three types of sites—HISD, other school districts, and nonprofit organizations—were equally likely to serve African-American, Asian/Pacific-Islander, and American-Indian youth (p<.05). In contrast, for Hispanic youth, HISD sites were more common, followed by other public schools, and then by nonprofits. For Anglo-Caucasian youth, nonprofits were still the least frequent, but other public schools served more Anglo youth than HISD sites.

There was no significant (p<.05) difference among the three types of sites in the monthly withdraws from the program.

HISD sites had significantly (p<.05) more children enrolled each month, more new enrollments each month, and higher average daily attendance than other school district sites, which in turn were higher than nonprofit organization sites.

No significant (p<.05) differences were found between the three types of sites in terms of the proportion of youth coming from moderate-income families. However, HISD sites were greater in the number of participants who were from low-income families, who were from female-headed families, and who received free/reduced-price lunch than other public school sites, and in turn greater than nonprofit sites.

Program quality, according to the independent assessment conducted by the ASAP office coordinator, was similar for all three types of sites. Significant (p<.05) differences were found in the category of administration, but later analyses revealed no substantive difference.
Recruitment/Participation Almost 12,000 children were served by ASAP on a monthly basis, an average of approximately 120 children per site, during the 2000–2001 year.

More than 55% of parents had one child in ASAP while close to 40% of parents had two or three children in ASAP.

Fifty-three percent of participating students were Hispanic, 41% were African-American, 4% were Anglo, and 2% were Asian. There were significantly (p<.05) more Hispanic and African-American students than other ethnicities.

Eighty-eight percent of ASAP participants were between the ages of five and twelve and enrolled in the first through eighth grades.

There were equal numbers of male and female children in ASAP.

Forty percent of program participants came from homes headed by single mothers.

About 35% of parents considered that their incomes were “inadequate” or “uncomfortable.”

Seventy-three percent of program participants were from low-income families and 77% received free or reduced-price lunch (83% qualified). Ten percent came from families of moderate income.

Less than 1% of participants were disabled.

There were an average of 54.5 participants in each activity offered.

Average daily attendance rates were approximately 64%.

Average monthly enrollment was at 97% of total capacity and withdrawal rates were only about 3%. Of the total monthly participants, 30% were new enrollments.

Monthly new enrollment was positively (p<.05) related to the number of participants from low- and moderate-income families, female-headed families, children receiving free/reduced-price lunch, and children from African-American and Hispanic ethnic backgrounds. The number of withdrawn participants was positively (p<.05) related to the number of participants from low-income families, female-headed households, participants receiving free/reduced-price lunch, and Hispanic youth.
Satisfaction Seventy-six percent of surveyed participants reported “definitely” liking ASAP, 20% answered “kind of,” and 4% answered “not at all” to the same question. Significantly (p<.05) more participants felt positively about the program than did not.

Sixty-nine percent of surveyed participants reported that they would “definitely” wanted to attend ASAP next year, 19% reported that they “kind of” wanted to, and 13% reported that they did “not at all” want to. Significantly (p<.05) more participants wanted to return to the program than did not.

Student satisfaction survey responses revealed that participants were predominantly satisfied with ASAP. All mean scores were significantly (p<.05) more positive than neutral.

Parent survey respondents indicated a high level of satisfaction with ASAP, usually answering “4” on a Likert 5-scale survey. Parents' overall rating of the program, as well as mean scores for all items, were significantly (p<.05) more positive than neutral.

More than 90% of parents reported that they would recommend the program to other parents.

Parents' positive perceptions of the functioning and offerings of ASAP were positively and significantly (p<.05) related to the number of their children in the program. Satisfaction and need were not significantly (p<.05) related to the number of children parents had in the program.

Parents of younger children were significantly (p<.05) more satisfied with the program than parents of older children.

Parents with lower family incomes were significantly (p<.05) more satisfied with the program than parents with higher family incomes.

Parents of African-American and Hispanic youth were generally more satisfied with the quality of the program. Parents of Asian children provided average assessments of ASAP.

Parents of American-Indian children were very satisfied with the overall program, but had reservations about program operation. Parents of Caucasian children were least satisfied with ASAP, of all ethnic groups.

Parents whose children would be with a private sitter or a relative in the absence of ASAP were more satisfied overall with the program than those who had other arrangements. In other words, ASAP was more satisfactory to those who had the greatest child care needs.

Participants were generally satisfied with the ASAP Partnership Fair, assessed with survey items falling into each of four categories: general quality, workshop instruction, vendor exhibits, and exemplary programs. Survey respondents predominantly chose “4” and “5” on a Likert 5-scale survey, with all mean scores significantly (p<.05) more positive than neutral.

Parent interviews revealed a theme that parents feel ASAP must be a priority for the City of Houston.
Staffing/Training Evaluators identified key factors critical to successful implementation of an after school program—best practices—one of which is dedicated teacher involvement.

In interviews, facilitators/teachers expressed concern over low salaries, about $20/hour for staff and $7–$12/hour for teachers' aids. School district policies determine compensation. Social workers and nurses are not eligible to be paid staff members. Also, staff mentioned difficulty recruiting and maintaining volunteers.
Systemic Infrastructure According to ASAP teacher surveys, the ASAP office was generally considered to be effective, with mean scores significantly (p<.05) above neutral.

Evaluators identified key factors critical to successful implementation of an after school program—best practices—one of which is strong program management and accurate reporting.


Summative/Outcome Findings

Academic Students attending ASAP scored significantly (p<.05) higher on the academic posttest in the areas of science and fine arts than student test takers in the comparison group. There were no significant (p<.05) differences in any of the other nine areas tested (reading, other language arts, mathematics, social studies, handwriting, physical education, health and safety, computers, and science lab), but posttest mean scores for program participants were all clearly higher than for comparison group members.

According to interviews with students, a theme emerged that students perceive that ASAP is contributing to their improved academic performance.
Family In interviews with parents, one theme that emerged was that parents depend on quality after school programs.
Prevention All mean juvenile crime variables for the period between August 2000 and May 2001, between 3pm and 6pm, were significantly (p<.05) lower than the three previous years, but analyses of covariance indicated that differences between those beats with ASAP programs and those without were not statistically significant (p<.05).
Youth Development Facilitators/teachers from 75 sites evaluated students using a Likert 5-scale survey. Predominantly, the teachers selected “4” and “5,” meaning that the teachers considered that as a result of participation in ASAP students had made great improvements in the cognitive, affective, and behavioral domains. The mean scores were all significantly (p<.05) greater than neutral (3).

Facilitator interviews revealed a theme that facilitators feel that time after school with students creates trust among students, facilitators, and parents. Many facilitators commented that increased time after school with students allows them to build stronger relationships with both children and parents.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project