You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼


Research Description

Overview and Components This study examined high-performing after school projects funded with grants from The After-School Corporation (TASC) to determine project characteristics that may have contributed to improvements in youth’s academic achievement. In a reanalysis of New York City student performance data, the study team identified 10 TASC projects whose after school participants showed significant 1-year gains in math and English/language arts/reading compared with peers in their schools who did not attend the TASC project. These 10 projects were then visited by researchers to identify common features of programming, staffing, and support systems.
Start Date 2004
Scope local
Type after school
Location urban
Setting public school
Participants elementary through middle school students
Number of Sites/Grantees 10 projects
Number Served Average enrollment across sites was 262 youth.
Study Details TASC provides grants to nonprofit organizations that partner with individual New York public schools to support school-based projects that aim to improve academic learning, promote healthy youth development, and reduce antisocial behavior. Under the TASC approach, after school services are provided through a partnership between a public school (known as the host school) and a local nonprofit organization with ties to the community served by the school. All students enrolled in the host school are eligible to participate in the after school project, which provides services from the end of each school day to approximately 6 p.m. Services are intended to supplement the learning experiences of the regular school day. Programming generally emphasizes academic enrichment, homework assistance, the arts, and recreation. The project’s objective is to combine community connections, youth interest, cultural resources, and specialized foci of selected nonprofit organizations with the academic focus, facilities, and access to students provided by public schools.
Funding Level $135,000
Funding Sources The study was funded by the United States Department of Education through a contract with the Southwest Educational Development Laboratory (SEDL).
Researchers Policy Studies Associates, Inc.
Research Profiled Shared Features of High-Performing After-School Programs: A Follow-Up to the TASC Evaluation
Research Planned PSA continues to conduct reanalyses of TASC data to deepen understanding the relationship between after school experiences and youth outcomes, including a current study of the effect of TASC participation in the middle grades on high school performance.
Report Availability Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high-performing after-school programs: A follow-up to the TASC evaluation. Washington, DC: Policy Studies Associates. Available at www.policystudies.com/studies/youth/TASC-SEDL.html or www.sedl.org/pubs/catalog/items/fam107.html


Contacts

Research Elizabeth Reisner
Policy Studies Associates, Inc.
1718 Connecticut Avenue, NW, Suite 400
Washington, DC 20009
Tel: 202-939-5323
Fax: 202-939-5732
Email: ereisner@policystudies.com
Program Lucy Friedman
President
The After-School Corporation
925 9th Avenue
New York, NY
Tel: 212-547-6951
Email: lfriedman@tascorp.org
Profile Updated October 24, 2006

Research Study: Shared Features of High-Performing After-School Programs: A Follow-Up to the TASC Evaluation



Research Description

Research Purpose To examine high-performing TASC projects to identify common characteristics, if any, that may contribute participants’ educational performance.
Research Design Quasi-Experimental and Non-Experimental: Researchers ranked TASC projects based on the differences in average gains in math and English/language arts (ELA) achievement from the 2000–2001 school year to the 2001–2002 school year between active participants (youth who attended TASC for at least 60 days and 60% of the days it was possible for them to attend in 2001–2002) and nonparticipants (youth who attended the TASC host school in 2001–2002 but never attended TASC). This analysis was limited to 76 TASC host schools with at least 25 students in both the participant and nonparticipant groups who performed below grade level in the 2000–2001 school year. Researcher asked TASC managers who work closely with the projects to comment on the status and strengths of the top 20 projects based on these ELA/math rankings. The TASC managers identified projects that were no longer operating, or where major changes had occurred in project quality, compliance with the TASC model, principal commitment to the project, or other factors that may have affected the project. Based on this feedback, the 10 most promising projects were invited to participate in the study, and all 10 agreed to do so. All 10 projects served elementary school students, and 6 also served middle school students. Researchers visited each of the 10 study sites for 2 consecutive days each to conduct observations and site coordinator interviews.
Data Collection Methods Interviews/Focus Groups: Site coordinator interviews focused on the project’s process and content features (e.g., goals, activities) and structural and institutional features (e.g., staffing, supervision, resources).

Observation: Researchers conducted up to 20 15-minute structured observations of after school activities at each site.

Secondary Source/Data Review: The New York City Department of Education provided student data for school years 1997–1998 through 2001–2002, including demographic characteristics, test scores, and school attendance.

Test/Assessments: Scores on statewide math and ELA tests administered in Grades 4 and 8 and on comparable citywide tests administered in Grades 3, 5, 6, and 7 were collected and analyzed.

During observations, researchers used Policy Studies Associates’ Out-of-School Time Observation Instrument (Birmingham, Pechman, Russell, & Mielke, 2005) to rate project activities on five key domains related to youth development: (a) youth-directed relationship building (e.g., youth are friendly to each other); (b) youth participation (e.g., youth are on-task); (c) staff-directed relationship building (e.g., staff use positive behavior management techniques); (d) staff strategies for skill building and mastery (e.g., staff communicate goals, purposes, and expectations); and (d) activity content and structure (e.g., the activity is well organized). Each indicator was rated on a scale of 1 to 7, where a 1 meant that the indicator was not evident, and a 7 meant that the indicator was highly evident and consistent.

Reference:
Birmingham, J., Pechman, E. M., Russell, C. A., & Mielke, M. (2005). Shared features of high-performing after-school programs: A follow-up to the TASC evaluation. Washington, DC: Policy Studies Associates.
Data Collection Timeframe Observation data were collected in spring 2005. Test score data utilized in the analyses were collected between 2000 and 2002.


Findings:
Formative/Process Findings

Activity Implementation Project activities occurred for almost 3 hours each school day in a safe environment that informally connected youth with a cross-age mix of peers and positive adult role models.

Project leaders, school officials, and parents ensured that youth finished some homework each day and that youth were exposed to a balanced array of experiences that promoted healthy overall development.

Weekly schedules included homework help, project-based activities, arts and crafts, performing arts, and recreation. Occasionally projects offered specialized academic support such as individual tutoring or small-group instruction. Projects offered three or four opportunities a week for youth to participate in varied arts and sports activities. Science and math activities were rarely included in the schedule.

For many participants, the after school project provided their first exposure to learning opportunities such as dance, music, or a field trip outside their neighborhood.

During project start-up, leaders developed clear and common expectations for participant interactions and defined the consequences when expectations were not met. Some projects hired project graduates to establish an atmosphere of mutual respect and collaboration.

Through sports and recreation activities, participants had time to run, play, interact in loud voices, and move freely with limited structure. For many participants, it was the first time each day that they could run freely in a gym or on a playground.

Each project created opportunities to build participants’ literacy skills through reading, storytelling, and writing activities. Younger children participated in shared reading time, in which they described to one another the books they had chosen by reading short passages or describing the story in pictures. Older participants wrote poetry, rehearsed plays, and designed newsletters and yearbooks. In addition, all 10 projects provided structure for literacy activities by using formal curricula.

Homework help occurred in small groups in 45 minute time slots, often managed by college students, with the assistance of teaching specialists or experienced after school leaders. Leaders were available to answer questions, help with assignments, and monitor participants in completing their homework, but rarely did they check homework for accuracy. At least 4 days per week, leaders oversaw small groups of participants working on homework. They typically had books and academic games available to those who did not have homework or had completed it early.

Projects provided numerous access points to the arts. For example, one project was structured around theme-based arts activities. Throughout the year, participants researched a theme related to New York City’s history and culture, then integrated their learning into a culminating performance through visual arts, drama, and music activities.

Researchers saw strong evidence that the majority of activities observed were well organized (75% were rated 5 or higher) and challenged youth intellectually, creatively, and/or physically (58% were rated 5 or higher).

Both academic enrichment and arts activities scored significantly higher than sports activities on the activity content and structure scale (p < .05). In addition, activities that were categorized as intentionally skill building had significantly higher scores on this scale than did activities that were not intended to be skill building (p < .05).

Arts activities often gave youth opportunities to practice and master new skills in preparing for performances or exhibitions. Dance, music, and drama enabled youth to express themselves in new ways and to create cultural connections with the community.

In the majority of activities, most youth were friendly to each other, demonstrating relaxed interactions (rated 5 or higher in 93% of activities); showed respect for one another, refraining from causing disruptions and considering each other’s viewpoint (86% of activities); were on task (84%); showed positive affect to staff, as demonstrated through friendly interactions (82%); and listened actively and attentively to peers and staff (76%).

On average, arts activities showed stronger youth relationships and participation than did homework activities.

Analyses showed significantly higher scale scores on the youth relationship building and participation scale in activities that did not target academic skills (p < .05).

On average, activities that targeted physical, artistic, interpersonal, or other types of skills had a significantly higher scale score on the youth relationship building and participation scale than activities that targeted academic skill-building (p < .05).
Parent/Community Involvement Coordinators regarded connections with families to be vital to their success, although few families had the discretionary time to become deeply involved. Most projects were staffed with parent coordinators who were the primary point of contact between the family and the project; often, they were also employed by the school as its parent coordinator.

At the beginning of the school year, staff in several projects met with families in orientation sessions in which parents received information about the project’s philosophy and approach and about resources that they could access through the project or project sponsor.

Site coordinators and group leaders maintained regular contact with family members who picked children up from the project, at which time they shared news of their children’s successes and of forthcoming events. Newsletters and flyers went home to announce parenting workshops, English as a Second Language classes for adults, and community-oriented service activities and to alert parents to participant performances and activities. These activities gave parents with limited time opportunities to participate to the extent they could. Several projects periodically wrote reports on participants’ behavior and performance that was provided to parents or to teachers to include in reports to parents.

Families were connected with the project sponsors’ social service resources, which were generally available to families at no cost, and the projects typically benefited from professional social workers, job counselors, and youth staff, often also at no extra cost.
Program Context/Infrastructure Projects adopted policies that established a tone of welcome, respect, and inclusion. The policies also established project attendance requirements, behavioral expectations, and the consequences for participants who did not meet the behavioral expectations.

The projects’ relationships with their partners built the foundations of their success and sustainability. In these partnerships, the sponsoring organization gave the site coordinator the autonomy and flexibility to manage the project day to day, while providing administrative and fiscal support. Site coordinators were then able to use their expertise to select activities and to make staffing decisions. Regular communication between the sponsor and the site coordinator, which often took the form of weekly updates, monthly reports, meetings, or phone conversations, kept both parties up to date.

Sponsors provided staff professional development. Some sponsors ran more than one after school project, and site coordinators drew on the experiences of their colleagues to solve problems within their own projects. Other sponsors developed extensive, in-house training that all new staff attended at the beginning of the year. Some experienced site coordinators shared their own knowledge by leading professional development sessions within and outside the sponsor organization.

Through their strong financial and managerial support, sponsors identified and secured resources to ensure sustainability and freed site coordinators to concentrate on creating thriving projects. Projects also benefited from the shared funding streams available to them through the sponsors’ resources.
Program–School Linkages Project leaders, school officials and parents were committed to linking after school activities to school-day priorities.

Site coordinators forged connections with host school staff. Strong partnerships between the project and the host schools helped determine whether a project ran smoothly or not. Essential ingredients of the most effective partnerships were mutual respect between the project coordinator and the principal, shared teaching and paraprofessional staff members, appreciation that students benefited from the project, and flexibility among schools’ teaching, custodial, cafeteria, and security staff.

All projects scheduled regular planning times with the principals of their host schools, but there was no standard for frequency of meetings or individuals involved. Six projects had a formal feedback mechanism for teachers to register concerns that arose when the project used classrooms; other projects promoted informal connections with the teachers in whose rooms they worked and with the project leaders who managed groups. All projects elicited information from the school about homework assignments.

Among the most successful strategies to align academic priorities across the school and after school projects were: (a) hiring a school day teacher to keep the project apprised of the school day; (b) using school day literacy and math materials to help focus after school academic support; (c) observing in classrooms and talking informally with schoolteachers about youth’s learning needs and behavior; (d) pooling resources to hire arts and recreation specialists for the school day and the after school project; (e) arranging for classrooms, libraries, manipulatives, and games to be available to support after school academic activities; (f) hiring school-day paraprofessional aides to coordinate youth’s records of academic progress between the school-day and after school project; and (g) using the school’s parent liaison to facilitate connections with the school and families.

Some projects allocated resources to provide school staff with beginning-of-the year “care bags” and thank-you gifts and to host joint celebration and appreciation days. Teachers received materials for use in their classroom; school staff received project T-shirts; and everyone school-wide was invited to come to all after school performances, celebrations, and parties.

Site coordinators valued the strategy workshops that school day teachers periodically conducted for project staff in some projects. “There’s nobody better to teach my staff how to do homework and math literacy help than the literacy and math coaches for the teachers at the school,” reported one coordinator. In addition to informing project staff about what was happening during the school day, teachers brought project staff’s feedback back to their colleagues.
Staffing/Training Project instruction was typically led by adult paraprofessionals or college students who were supervised by experienced project coordinators or certified teaching specialists.

In the majority of activities observed across the projects, staff tended to use certain common instructional strategies to promote skill building and mastery. Specifically, project staff: (a) communicated the goals, purposes, and expectations of the activity in which youth were engaged (76% of activities); (b) assisted youth without taking control of the activity, for example, by coaching or employing scaffolding techniques to help youth gain a better understanding of a concept or complete an action on their own (66%); and (c) verbally recognized youth’s efforts and accomplishments (58%).

In the majority of skill-focused activities, staff communicated activity goals clearly (88% of activities), assisted youth without taking control (79%), and verbally encouraged youth (69%). Also in a majority of these activities, observers rated the following instructional strategies as a 5 or higher: (a) project staff challenge youth to move beyond their current level of competency through constructive feedback (59% of activities); and (b) project staff employ two or more teaching strategies to engage youth, including a combination of direct instruction, coaching, modeling, demonstrating, or other strategies (55%).

In activities where instructional strategies promoting skill building and mastery were evident, staff provided specific individual feedback and encouragement to youth. For example, in a homework help activity at one site, the group leader regularly praised youth who were concentrating on their work and reminded others to stay on task. The staff also emphasized accuracy and checked completed homework. Group leaders coached individual youth in math and encouraged high standards in expository writing.

On average, staff in academic and visual/performing arts activities used skill building instructional strategies significantly more frequently than did staff in sports activities.

Where sites could budget for specialists to lead art activities, youth tended to benefit from a greater degree of skill-building and mastery-focused instruction. Observations showed that activities led by specialists had a significantly higher score on the skill building and mastery scale than the score for activities that did not include these specialist staff.

Projects’ successes pivoted on the relationships that staff nurtured among participants. Coordinators measured their success by the “feeling of calm, respect, openness, and honesty” conveyed in activities and in informal exchanges. They also judged their success by the degree to which youth acted and felt like a community, in the respect they gave one another, and in the relationships staff developed with families and community members.

Coordinators talked often with staff about “the importance of making every child and parent [know] they have a stake in what we do.” They described “reteaching everyone” to “drop whatever baggage you’re coming in here with.” Coordinators emphasized to staff the importance of communicating often and openly with participating youth. When behavioral issues arose, the first response was to “have a conversation and find out what’s going on,” then encourage youth to reflect on their behavior and its consequences with project leaders or staff with counseling training.

Periodic focused training and conversations during staff meetings taught staff to conduct conflict resolution activities and to strengthen peer relationships using written curricula. Site coordinators also brought in social workers and experienced youth specialists in some instances to conduct relationship-building activities for staff and youth.

Observations revealed that staff: (a) were equitable and inclusive, encouraging the participation of all youth (90% of activities); (b) used positive behavior-management techniques, such as setting appropriate limits and communicating clear expectations for behavior (83%); (c) showed positive affect toward all youth, using a caring tone and positive language (82%); and (d) attentively listened to and/or observed youth, paying attention as they completed a task and responding to what they said (82%).

Academic enrichment activities had significantly higher scores on the staff–youth relationships scale than did homework help activities (p < .05). Similarly, in activities that were skill focused (e.g., dance or drama rehearsals), observers saw significantly stronger staff–youth relationships than in those that were not skill focused (e.g., games or open recreation time; p < .05).

High-performing projects were typically led by site coordinators with 5 or more years of experience working for the grantee organization or for similar organizations; all 10 had strong ties to the communities where they worked. Virtually all of the coordinators used such phrases as, “These are my children,” or “We’re family here,” when referring to participants.

Coordinators had a vision of what they were trying to accomplish, and they set goals and hired staff to achieve that vision. They understood that they were charged with both promoting positive cognitive development and establishing an emotional and developmental safety net for youth.

Many projects began the year with intensive mandatory orientation sessions during which staff met and interacted with one another, learned project policies and procedures, contributed to planning activity schedules, and discussed behavior management techniques. According to coordinators, these sessions were a crucial first step in the challenge of bringing activity leaders, security staff, principals, and teachers alike on board with the idea that children are at the front and center of the project’s goals.

Training continued throughout the year in staff meetings and in-service sessions on using curricula or managing children with developmental sensitivity. Coordinators met the challenge of scheduling meetings with part-time staff by extending the work day 1 hour, reducing the project day for youth by 1 hour, or holding meetings on days when the project was not in session. Coordinators also alerted staff to trainings offered by external vendors. Staff who volunteered to attend such trainings were usually paid a stipend to attend.

Long-time staff mentored new staff, and older staff mentored younger staff to develop common strategies to manage behavior and to motivate participants through hands-on activities. In sites with assistant coordinators or certified teaching personnel, trained staff were paired with less-experienced staff to learn how to plan and write lessons and to work with teaching curricula. Some projects used submission of lesson plans to improve staff work, while others used formal and informal staff evaluations to improve mentoring.

Leaders used informal networks to identify potential new staff. Coordinators reported a very high staff return rate, as much as 75% to 80% per year. They believed the care they took in hiring staff, despite part-time hours and low wages, contributed to high retention rates. Coordinators turned to community members, including paraprofessionals who worked in schools and people who worked with project sponsors to identify potential staff.

Coordinators were intuitive about which staff were the right fit for their project. As one coordinator said, “I can read between the lines when I am hiring someone… the way they look at the kids, if they notice the work on the wall.” They also favored staff with special interests and talents, such as coaching, arts and crafts, and so on. They looked for “high-energy people who have fun with kids.” Coordinators regularly described their work as a calling, and they sought staff who came with commitment to hard work and a willingness to grow.

Each project was led by a small management team, typically including the site coordinator and one or two assistants, which hired and trained staff and managed day-to-day operations.

Visiting artists and recreation specialists brought expertise in arts and sports to vary and strengthen the quality of project offerings. In some projects, staff with special skills in these areas were hired as group leaders who took on special initiatives within the project. Other projects hired school-day teachers who wanted to teach something new and creative, and some made connections with past participants and hired them to assist in the project.

Hourly wages were typically as follows: $7–10 for high school students and inexperienced college students, $10–12 for school aides, paraprofessionals, and older college students, $20–30 for arts specialists (who were not certified teachers), and upwards of $35 per hour for certified teachers (as dictated by their contract, although they sometimes volunteered to work for slightly less). Projects compensated for low salaries by providing mentoring and in-service training and by offering a collegial community.

In the face of low salaries, small salary increases, and advancement only with additional schooling, site coordinators relied on the power of their relationships to maintain stable, committed staffs. Communicating praise and encouragement went a long way toward building collegiality and commitment to the project.

Coordinators included staff in project planning in a number of ways. In addition to the planning done in staff meetings, some site leaders formally and informally surveyed staff so they could anonymously report their thinking about project quality issues.

Coordinators used social events to strengthen informal ties. They celebrated special occasions with cakes and gifts. They periodically hosted staff potlucks meals or convened over pizza, and sponsors included staff in neighborhood celebrations and holiday parties.

 

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project