You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview Located in an inner-city north-central Philadelphia neighborhood, the Cooke Middle School After School Recreation Program (CASP) complements the school's more academic-based programs by offering activities designed to promote students' physical, emotional, and social well-being. Offered during the evenings so that all of the school's students (whether they participate in an academic program or not) may participate, CASP's goals are to: (1) provide a safe, structured, adult-supervised environment that encourages positive adolescent development for as many of the school's students as possible; (2) provide opportunities for exercise and to develop fitness skills and wellness; (3) offer extracurricular activities that encourage the development of students' social, emotional, intellectual, and physical skills; and (4) teach children how to get along in a nonthreatening environment, where they have the chance to interact with teachers outside of the normal educational setting.
Start Date fall 1999
Scope local
Type after school
Location urban
Setting public school
Participants middle school students
Number of Sites/Grantees one
Number Served 126 in 1999–2000
Components The program operated Monday through Thursday evenings from 5pm to 7pm. Activities included sports/physical fitness, art, dance, board games, reading, and homework. Facilities at Cooke Middle School, including the gymnasium, a fitness center, and a classroom, were used for these activities.
Funding Level N/A
Funding Sources School district and 21st Century Community Learning Center funds


Evaluation

Overview CASP's evaluation contains two components, a process evaluation and an outcomes evaluation. The process evaluation looks at how the program is implemented, how closely program activities reflect the program's goals, and how the program could be strengthened given limited resources. The outcomes evaluation uses a random assignment design to study the program's impact on students' time use, attitudes, safety, and academic achievement.
Evaluator Sherri C. Lauver, University of Pennsylvania
Evaluations Profiled Assessing the Benefits of an After-School Program for Urban Youth: An Impact and Process Evaluation
Evaluations Planned Further evaluations are planned as part of the larger scope of CASP's 21st Century Community Learning Centers funding.
Report Availability Lauver, S. C. (2002). Assessing the benefits of an after-school program for urban youth:
An impact and process evaluation. Philadelphia: Author.


Contacts

Evaluation Sherri Christine Lauver, Ph.D.
Center for Educational Evaluation and Technical Assistance
Institute for Education Sciences, U.S. Department of Education
80 F. St., N.W., Suite 308
Washington, DC 20208
Email: sherri.lauver@ed.gov
Program Edward Davis
Program Director
Jay Cooke Middle School
1300 West Louden Street
Philadelphia, PA 19141-2699
Profile Updated August 7, 2003

Evaluation: Assessing the Benefits of an After-School Program for Urban Youth: An Impact and Process Evaluation


Evaluation Description

Evaluation Purpose To determine whether an after school program administered by an urban public middle school, with limited funding support and technical assistance, can make meaningful improvements in academic and social outcomes for young adolescents (ages 10–14) living in poverty-impacted neighborhoods. More specifically, the CASP evaluation addresses three questions: (1) how is Cooke Middle School's after school program implemented and how closely do program activities reflect the program's goals? how could the current program model be strengthened given limited resources? (2) what lessons does this program offer other urban school administrators who would like to implement and sustain an after school program? and (3) what impact does the after school program have on the students' time use, attitudes, safety, and academic achievement?
Evaluation Design Experimental, Quasi-Experimental, and Non-Experimental: The study sample includes 227 students who applied to the after school program and were subsequently randomly assigned to the program (N=124) or to a control group (N=98). Five students were deleted from the dataset because they moved out of the district. A comparison of the two groups revealed no statistically significant differences between the two groups in terms of grade-level, age, race, gender, household composition (two-parent family vs. single-parent family vs. living with a guardian), educational attainment of parent/guardian, and language spoken at home. The program group was significantly more likely to have working parents/guardians (79.8% working vs. 57.1% working, p<.01) and to have more total people (and children) in their households (p<.05). The only occasion in which random assignment was not followed was when a student with siblings at Cooke was randomly assigned to the program, in which case the non-assigned sibling was also included in the program group (so that parents would not have to worry about finding alternate care). All analyses were completed on both the full sample and the sample without the 18 siblings who were not randomly assigned. There were no differences in significance levels or regression coefficients between these two samples, so for all the results presented here, the full larger sample is used.

The research sample ranged in age from 10 to 15 years and 59% were in grades seven and eight. Approximately 83% of the sample was African American and 60% was male. Forty percent of the research sample's households consisted of two-parent families.

The evaluator also conducted a separate quasi-experimental analysis looking at how program attendance related to the various outcome measures. After assigning each student a “dosage level” based on the number of days students attended the program (with control students assigned a dosage level of zero), the evaluator looked at whether this dosage level was significantly related to the outcomes of interest. The evaluator cautions, however, that any such relationships may be due to students' self-selection into higher attendance levels.

The evaluator conducted a “power analysis” in order to estimate the minimum detectable differences that could be detected in the sample. This power analysis revealed that the sample could detect grade point average changes as small as about .3 points (on a 4.0 scale), SAT-9 changes as small as eight to ten percentiles, attendance changes as small as about three days, and activity participation changes as small as about 20 minutes.

In addition to the experimental and quasi-experimental components, a non-experimental process evaluation component consisted of multiple site visits and focus groups over the course of two school years (1999–2000 and 2000–2001).
Data Collection Methods Interviews/Focus Groups: Three focus groups were held with a total of 20 program participants in order to tap into attitudes and feelings about the program, as well as suggestions for future years. In-depth interviews were conducted with all seven staff members and the program director to assess attitudes about program success and challenges faced during program operations. Interviews were again conducted at the end of the second year to document program strengths and weaknesses.

Observation: Site visits focused on documenting program environment, activities, staffing, interactions among participants and staff, student engagement in program activities, students' experiences and satisfaction with the program (including perceived benefits of involvement), strengths and weaknesses, facility use, and management and administration. During 1999–2000, 40 of 64 total program days were observed. Informal site visits continued into the second year. As participant observers, the researchers assisted with program tasks and developed friendly, but not overly deep relationships with staff and students. The researcher's role was to observe the natural order and duration of activities rather than determine what activities took place and how long they would last.

Secondary Source/Data Review: Academic records, including report card, and attendance data were obtained from the School District of Philadelphia's Office of Pupil Information for the 1998–1999 and 1999–2000 academic years.

Surveys/Questionnaires: Parents completed a baseline questionnaire as part of the program application and informed consent procedure. This questionnaire asked about personal and demographic information. In June of 2000, sample youth completed a follow-up survey regarding background characteristics, activity involvement, self-esteem, conflict resolution skills, school and neighborhood safety, perceptions about school, time spent on various activities and homework, and future expectations (220 of 227 youth completed this survey). End-of-year parent surveys were sent to parents of children who had attended CASP at least 20% of the time. The response rate was 67% (31 of 46), and the survey asked about their satisfaction with the program and their perceptions of program quality and outcomes for their children.

Tests/Assessments: SAT-9 achievement test data were obtained from the School District of Philadelphia's Office of Pupil Information for the May 2000 test.
Data Collection Timeframe Data were collected primarily during the 1999–2000 school year, although program observations and informal interviewing continued into the 2000–2001 school year.


Findings:
Formative/Process Findings

Activity Implementation CASP participants were able to choose from among several activities during each evening session. During year 2, the program director decided to add more structure and students were told which activities were available at the beginning of the session and were allowed to switch activities after the first hour.

The gym was always open to participants and was a popular site for many of the boys (and some girls) who enjoyed playing basketball. Girls also frequently played double Dutch jump rope or socialized in the gym.

Arts and crafts were planned for Monday evenings, but were often cancelled due to the two art teachers' frequent absences and lack of challenging or well coordinated activities. Arts classes became more regular and more structured during year 2, but trailed off later in the year as materials ran out.

On Tuesday and Thursday evenings, many students participated in a dance group or worked out in a fitness center containing weight machines, treadmills, stationary bicycles, and other fitness equipment.

The dance group was found to be well structured and resoundingly popular. The group culminated in a performance during the school day at the end of the school year that included five choreographed dances (one involving sign language) and poetry readings by African-American poets. One feature of the dance group found to be effective was its clear structure and expectations for participants. The participants were given handbooks outlining the instructor's expectations and included written instructions and illustrations of the dance steps and sign language that would be learned. Approximately 20 students participated in a Spiritual Dance group regularly held in the auditorium on Tuesday and Thursday evenings. This activity was so popular that many of the students chose to observe the dance class rather than participate in other activities.

A game room was available sporadically, if there was an additional staff member available to supervise it. The game room included checkers, cards, UNO, bingo, chess, monopoly, etc. Students could also use this space to work on homework or draw.

Students were offered a snack at the end of the evening before going home.

The biggest addition to CASP in year 2 was the availability of the computer lab. Students used the Internet and played computer games, while a teacher oversaw lab activities in order to monitor the computers' usage.

Two staff members leaving CASP resulted in the termination of two popular programs initiated in year 1, Martial Arts and the dance class. Researchers observed that many of the girls who had participated in the dance class stopped coming to CASP after the instructor left. Due to popular demand, a new dance instructor was hired during year 2.

Almost 40% of students listed basketball as their favorite activity, while about a third enjoyed song and dance activities the most. Basketball and dance came up most frequently as students' favorite activities in the focus groups as well.

When surveyed, 62% of regular program participants wished for a greater choice of activities. During focus groups, several students mentioned their desire for courses in drawing, drama, and musical instruments. Others expressed a desire for more variety in sports activities.
Parent/Community Involvement More than 95% of caregivers agreed that they had been well informed about program activities.

A majority of parents (87%) agreed that the program's evening meeting time worked well for their family's lifestyle and schedule.
Program Context/Infrastructure The teachers and administrators at Cooke Middle School became increasingly concerned about students' academic achievement, safety, and vulnerability to high-risk behavior. In response, they undertook a number of initiatives to enrich the school's responsiveness to student needs. Such efforts were recognized by Education Week, which selected them as one of four middle schools that “have demonstrated their ability to meet the social and psychological needs of their students as well as provide a rigorous academic regiment.”¹ Only three years prior, the school district targeted Cooke Middle School to receive outside academic assistance due to its students' particularly low achievement test scores.

Approximately 85% of Cooke's 925 students were eligible for a free/reduced-price lunch (which is a common proxy for low socioeconomic status). In contrast, 76% of the students in the Philadelphia School District were eligible for the free/reduced-price lunch.

Ninety-nine percent of Cooke Middle School's students were of racial/ethnic minority status. Nearly 80% were African American, 7% were Latino, and 12% were Asian American.

Cooke students were performing in the bottom 25% of the state's students in reading and math. They also scored in the bottom 25% of students when compared to students of a similar income level.

The typical class size at Cooke Middle School was larger than the state average. More than 50% of the classes had 30 or more students, compared to 3% of elementary classes and 8% of classes in secondary grades statewide.

The neighborhood surrounding Cooke Middle School is similar to many other urban neighborhoods in large cities. The neighborhood consists mainly of row homes (many with broken or boarded-up windows) and a few small convenience stores and sandwich shops. There is a good deal of litter in the streets and sidewalks and graffiti is prevalent. The school is approximately two blocks off a major thoroughfare, where businesses, fast-food restaurants, shopping centers, and grocery stores are located.

According to the 2000 Census, the population in the neighborhood community is 80% African American, 10% Asian, 4% Latino, 3% white, and 3% other races. In the 1990 Census, the median household income was approximately $20,000, and 26% of the surveyed persons lived below the poverty level. Sixty percent of adults 18 years and over had at least a high school degree or GED.

Almost 97% of parents and guardians reported that the program was a positive and safe place for their child to be each evening.

¹ Manzo, K. M. (2000, October 4). Finding their voices. Education Week, 35-37.
Program-School Linkages Governance and use of space proved challenging for Cooke's program staff. The space allocated to the students in the after school program met only their basic needs. The students were allowed access to only portions of the first floor of the building and, since the library and computer room were housed on the second floor, they were inaccessible to the program. As opening the second floor would have required an additional security guard, this would require more funding. Furthermore, only one of the two gyms in the building was available for recreation use.

In addition, there was no storage space dedicated for the program's use. Complicating matters further were day teachers who were reluctant to offer their classrooms for evening use. During interviews, several staff members complained of the lack of a dedicated space to store art supplies, games, and a boom box for the dance group. Staff members were forced to keep materials in the trunks of their cars to ensure they would be available during after school activities. Art supplies were locked in one particular teacher's classroom, which other program staff were unable to access when this teacher was absent.

During year 2, the program staff had somewhat better access to space, most notably the school's computer lab. Access to this space also came with its own problems, however. Staff members had to be extra careful in making sure that students handled the computers with care.
Recruitment/Participation All 925 Cooke students were invited to apply to participate in CASP during the first year, of whom 227 applied for the program.

On a typical evening at the program, about 30 students participated and three or four staff members (all Cooke teachers/staff except for the security guard) monitored the activities and the hallways.

The research team worked with the school and program staff to recruit students. Signs were posted throughout the school and announcements were made on the school public address system. The research team handed out flyers, answered students' questions during lunch hours, and set up an information table during parent-teacher conferences. A question and answer session was held during a school meeting to explain the program and the concomitant research study. A few days before applications were due, the research team visited every classroom to remind students of the program and hand out applications to students who still needed one. In addition, the program director gave classroom teachers information about CASP and asked them to inform students about the program.

The reasons students indicated they wanted to apply to the program were: (1) to hang out with friends (43%); (2) to be involved in sports and recreation (34%); (3) interest in arts and crafts (20%); (4) to “feel like they belonged someplace” (15%); and (5) to be eligible to attend a roller-skating party offered as an incentive for students to apply (15%).

Program attendance was lower than expected. Twenty percent of the program group never attended the recreation program and another 23% attended less than six of the total 64 program days. Forty-two percent attended between 6 and 31 days and the remaining 15% attended more than half the time.

Given the lower than expected attendance, researchers made phone calls to parents of 71 program group participants to ask about their interest in having their child participate. Twelve of these students were no longer interested due to other responsibilities/activities, homework, safety and transportation issues, hours of operation, or lack of interest. Another 46 students expressed interest in participating, but could not commit to regular attendance due to safety/transportation issues, other activities, or another unspecified reason.

CASP attempted to combat low attendance through an incentive program, whereby students attending at least 75% of eligible program days each month received a free movie ticket for their efforts. The evaluators found, however, that this incentive was popular among only a small group of students, and that attendance continued to decline from about 30% of capacity in February to a low of 17% in May.

In the second year of CASP, 230 students applied to participate. Given the previous year's low attendance, all 230 were granted permission. About half of these were “regular” attendees, attending CASP at least 30% of the time. About 35 students participated on a typical night, with a high of about 60 students and a low of about 15.

Three factors were found to be related to higher program attendance: (1) seventh graders were more likely to attend than fifth, sixth, or eighth graders (p<.05); (2) children of working parents were more likely to attend than children of non-working parents (p<.05); and (3) the more children there were in the participant's home, the more likely they were to attend (p<.10).
Satisfaction Almost one quarter of the regular participants were not happy with program hours (5pm to 7pm) and one-fifth of the students were dissatisfied with the choice of days.

Almost all of the students looked forward to program activities and would recommend the program to their friends.
Staffing/Training Recruitment challenges forced the recreation program to begin several months after the program's intended start date during year 1. Seven teachers, two security guards, and two volunteers served the program. However, most of these people participated only one night each week. Furthermore, staff were often absent on their scheduled days.

During year 1, staffing challenges posed the largest obstacle to the success of the after school program. Recruiting was problematic due to the fact that it required teachers to work a 12-hour day. In addition to the long working hours, some teachers had not been paid for their work in the program the previous year because the district had not disseminated these funds to the school.

As a result of problems during year 1, the program director hired only about half of the staff members back the next year. Except for the new dance teacher (who was a community member), all of the newly hired staff members were teachers at the school. During year 2, there were a total of 10 staff members, including six teachers, two security guards (one of whom also acted as a teaching assistant), a teaching assistant, and a dance teacher. Several staff members worked two or more evenings per week and three staff members worked every evening. There appeared to be great improvements in staff attendance during year 2. Researchers generally observed the presence of between five and seven staff members, including the security guard monitoring the front entrance.

A certified physical education teacher staffed the fitness room in order to monitor students' use of the fitness equipment.

The evaluator found that most program staff earned students' respect by “leading with kindness.” They set guidelines for behavior that would ensure safety while allowing for youth's autonomy.

Youth were found to be happy with their relationships with staff. In the follow-up survey, 87% of the regular attendees indicated that “I get attention from the teachers and staff and they care about me.”

The volunteer dance instructor was found to be effective, in part because she had a similar background as many of her students. The dance group students saw her as a role model and someone who had managed, despite a childhood living in an “at-risk” environment, to attend a nationally reputed university. The devotion of her students was evidenced by the fact that six of her dance students crossed the city on two subways and a bus to attend her college graduation ceremony.


Summative/Outcome Findings

Academic Participation in CASP had no measurable impacts on academic performance or standardized test scores as measured by GPA and SAT-9 standardized test scores.

CASP participation showed no measurable impact on students' school attendance.

Program students demonstrated significantly higher levels of time spent on homework each week (p<.05) than control students.

Eighty percent of parents agreed that the program motivated their children to complete their homework and a similar number (81%) agreed that their children were more enthusiastic about school because of the recreation program.

For the program “dosage level” analysis, students with higher levels of program attendance were significantly more likely to attend school (p<.05) and spend time on homework (p<.05) than were students with lower levels of program attendance. Dosage level was unrelated to students' GPA and SAT-9 standardized test scores.

A very large percentage of students (close to 90%) indicated that their participation in the program positively impacted their opinions regarding school attendance.
Youth Development There were no statistically significant program impacts found for time spent on artistic activities or in exercise or sports.

There was a statistically significant program impact for time spent on strength training activities at least one hour per week (p<.01), with 80% of the program group spending time in these activities as compared to 61% of the control group.

There were no measurable program impacts found for program students' in-school behavior.

Participation in CASP had a significant impact on students' aspirations for further education (p<.01), as program students were significantly more likely to want to attend college or job training after high school.

CASP participation was not found to be related to students' time spent watching television or the amount of time they spent in self-care (time spent at home without an adult present).

Over 60% of youth stated that they had made four or more new friends and another 25% had made at least one to three new friends as a result of their participation in CASP.

Several staff members observed that CASP seemed to have a beneficial effect on many students' social skills. For example, CASP's dance teacher noted how one girl came to the program “so shy that she couldn't talk above a whisper,” but “before I knew it, she was up on that stage dancing and smiling and having a great time! … I think the dance group did something really special for her.”

Over 80% of parents and guardians agreed with the statement “The program has helped my child improve their overall behavior.” Similarly, more than 90% felt that the program helped their child increase self-confidence in areas other than academics.

Ninety-four percent of caregivers believed that the program helped their child to learn a new skill.

In the “dosage level” analysis, students with higher levels of program attendance were significantly more likely to spend one or more hours each week on enriching activities such as dance, art, or music (p<.05); on exercise (p<.10); and on strength training (p<.01) than were students with lower levels of program attendance.

Dosage level was not related to students' in-school behavior, college aspirations, time spent watching television, or time spent in self-care.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project