Search Publications | The Evaluation Exchange | FINE Network | OST Database

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

A Profile of the Evaluation of Save the Children Web of Support Initiative

Program Description

Overview	The Save the Children Web of Support Initiative promotes quality out-of-school time programs for children ages 5–18 by focusing on three desired outcomes: constructive activities, caring adults, and safe places. Save the Children provides resources and assistance to community-based organizations, including school districts, to operate both school-site and community-site after school programs.
Start Date	1997
Scope	national
Type	after school, summer/vacation
Location	urban, rural

Setting	public schools, community-based organizations
Participants	kindergarten through high school students
Number of Sites/Grantees	over 100 Rural Community Partners and six Urban Collaboratives
Number Served	approximately 7,000–9,000 youth
Components	The Web of Support Initiative provides support in three areas: funding and resources, leadership training and development to improve program practices and program quality, and technical assistance and training in how to implement and use outcome evaluation as a framework for program improvement. Rural Community Partners are typically single local nonprofit agencies serving youth in rural areas. These Partners are located in rural areas in the Southwestern, Appalachian, and Eastern regions of the United States. Urban Collaboratives typically consist of two or more nonprofit organizations working together to serve at-risk youth in low-income urban environments. Each Urban Collaborative has a Facilitator who assists with the coordination of the Collaborative. Collaboratives are located in or near large metropolitan areas. As part of the Web of Support initiative, Save the Children sought to develop the leadership capabilities of youth involved in the programs. Save the Children provided three different types of leadership development experiences for youth: the Save the Children Youth Summit, youth internships sponsored by Save the Children through Rural Community Partners and Urban Collaboratives (e.g., youth working as interns for the youth-serving nonprofit organizations), and other leadership opportunities offered in their communities by Partners and Collaboratives.
Funding Level	$32 million (over three years)
Funding Sources	Save the Children ($26 million over three years), the David and Lucile Packard Foundation ($3 million over three years), Denny's Family Restaurants (up to $1 million per year), and Irvine Foundation (Central Valley, California, expansion efforts)
Other	The National Institute on Out-of-School Time (NIOST) provides training and technical assistance to Rural Community Partners, Urban Collaboratives, and Save the Children regional office staff to improve the quality of out-of-school time programs. Aguirre International provides evaluation training and technical assistance to Save the Children regional and home office staff, Rural Community Partners, and Urban Collaboratives on how to evaluate programs and activities. The evaluation training uses an empowerment evaluation approach. Professional consultants (coaches) provide Train-the-Trainer workshops, on-site assistance, and phone assistance to staff. Annual evaluation trainings are also held in each region.

Evaluation

Overview	The Web of Support evaluation has two parts: (1) individual outcomes-based evaluations at the site level and (2) an initiative-wide evaluation. In the first part of the evaluation, each Rural Community Partner and Urban Collaborative conducts individual outcomes-based evaluations annually, with technical assistance from Aguirre International. They are required to examine the three components of a comprehensive out-of-school time program of the Web of Support Initiative: constructive activities, caring adults, and safe places. In keeping with the empowerment model of evaluation, the data collection methods used by Rural Community Partners and Urban Collaboratives vary. The purpose of the second part of the evaluation, the initiative-wide evaluation, is to examine three types of outcomes: child/youth outcomes, Rural Community Partner and Urban Collaborative outcomes, and Save the Children–U.S. programs outcomes. These evaluations are conducted annually, and the third-year evaluation looked at findings across the first three years in addition to that year's findings.
Evaluator	Aguirre International
Evaluations Profiled	1997–98 Web of Support Initiative wide evaluation. Annual report. 1998–99 Web of Support Initiative wide evaluation. Annual report. 1999–2000 Web of Support Initiative wide evaluation. Annual report. 2000–2001 Web of Support Initiative wide evaluation. Annual report.
Evaluations Planned	N/A
Report Availability	Terao, K. L., Morell, L. C., Stevenson, C. L., & Sloane, K. J. (1999). 1997–98 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group. Terao, K. L., Morell, L., & Stevenson, C. (2000). 1998–99 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group. Terao, K. L., Morell, L., & Stevenson, C. (2001). 1999–2000 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group. Terao, K. L., Morell, L., Stevenson, C., & Moulton, J. (2002). 2000–2001 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group.

Contacts

Evaluation	Ken Terao Evaluation Specialist 480 East 4th Avenue, Unit A San Mateo, CA 94401-3349 Tel: 650-373-4900 Fax: 650-348-0260 Email: kterao@aiweb.com
Program	Leigh Higgins Save the Children Federation Inc. U.S. 54 Wilton Road Westport, CT 06880 Tel: 203-221-4000 Fax: 203-221-4082 Email: lhiggins@savechildren.org
Profile Updated	August 19, 2004

Evaluation 2: 1998–99 Web of Support Initiative Wide Evaluation: Annual Report

Evaluation Description

Evaluation Purpose	To examine Year 2 of the initiative and three types of outcomes related to the initiative: child and youth outcomes, Rural Community Partner and Urban Collaborative outcomes, and Save the Children–U.S. Programs outcomes.
Evaluation Design	Non-Experimental: Information about child and youth outcomes was obtained through review of evaluation reports produced by each Rural Community Partner and Urban Collaborative. Evaluators conducted additional data collection to examine Rural Community Partner and Urban Collaborative outcomes and Save the Children–U.S. Programs outcomes.
Data Collection Methods	Document Review: Data on child and youth outcomes were gathered through content analysis of Rural Community Partner and Urban Collaborative outcome evaluation reports. These evaluations were conducted between October 1998 and September 1999. Surveys/Questionnaires: Youth involved in the youth leadership activities were asked to complete a survey to gather information about their knowledge and leadership skills and about internship experiences and expectations. Surveys were administered to Rural Community Partner members to gather information about the degree to which Rural Community Partners were collaborating, their skills in developing quality out-of-school time programming and internal evaluations, and the level of resources they were able to access because of their involvement in the Web of Support. Surveys of Urban Collaborative members gathered information about the effect the Web of Support had had on youth, their individual organizations, and local communities, the way in which internal evaluation results had been used, the value of having a facilitator, and the elements that worked and did not work well in their Collaborative. Surveys of Urban Collaborative Facilitators gathered information on the level of collaboration between partner organizations; the resources Collaborative Partners had increased access to; the effect the Web of Support had on youth, the Collaborative, and local communities; and the extent to which organizations in their Collaborative increased their evaluation skills. Surveys of Save the Children home and regional office staff measured their understanding of how to provide training and technical assistance to Rural Partners and Urban Collaboratives, the effectiveness of NIOST and Aguirre International's training methods, the technical assistance they and their Partners need in the coming year, and the impact the Web of Support has had on Save the Children.
Data Collection Timeframe	Data were collected between August and November 1999.

Findings:
Formative/Process Findings

Activity Implementation	Partners provided tutoring, recreation, computer training, leadership development, cultural enrichment, and other innovative programs designed to meet the needs of children in their community. Leadership development activities included an internship program, mentoring, facilitating activities for children, attending trainings and workshops, and delivering presentations at community meetings. Internships provided youth with the opportunity to be a part of youth groups working for change or to plan and participate in projects that focused on community service and outreach efforts.
Parent/Community Involvement	All Save the Children staff indicated their Community Partners demonstrated leadership skills by working with local community leaders to improve services for youth. Urban Collaboratives demonstrated leadership qualities by sharing staff, volunteers, interns, and office space within their Collaboratives. Rural Community Partners worked with other local organizations to provide academic, cultural, recreational, and other services in their after school programs.
Program Context/Infrastructure	Thirty-six Rural Community Partners and three Urban Collaboratives reported that they completed structural or landscape maintenance upgrades at program sites so that youth had a safe place for programs and activities. Partners reported that they provided youth with space for activities, comfortable atmospheres, safety precautions, and healthy snacks.
Systemic Infrastructure	Rural Community Partners received training and technical assistance in program improvement, community and leadership development, and mobilizing resources from NIOST. Urban Collaboratives received training and technical assistance in program improvement, youth leadership development, and collaboration from the home office. Most Rural Community Partners and Urban Collaboratives surveyed reported that the training they received from NIOST and Aguirre International was “helpful” or “very helpful.” Eighty-two percent of regional and home office staff interviewed reported that they gained the skills to provide technical assistance to Community Partners (rural and urban) on improving their programs. Ninety-one percent of the regional and home office staff said that they gained the skills to provide technical assistance in helping Community Partners with their evaluation efforts. Eighty-two percent of the regional and home office staff surveyed reported that the Train-the-Trainers model was “very effective” in helping them to become coaches for Rural Community Partners or Urban Collaboratives.

Summative/Outcome Findings

Academic	Of the 1,080 students evaluated in tutoring/homework assistance programs, 80% maintained high standards or showed improvements in one or more of the following: grades, homework completion, study habits, or other measures of academic success. Of the 1,987 youth evaluated in reading programs, 60% exceeded or met expectations in reading ability, comprehension, or interest.
Community Development	Ninety-two percent of Rural Community Partners reported that their community was more aware that Save the Children provided services in their communities during the past year. Sixty-seven percent of Urban Collaborative facilitators reported that their local community was more aware of Save the Children's contributions to their neighborhoods during the past year.
Systemic	Seventy-four percent of Rural Community Partners reported collaborating with other Save the Children Community Partners during Year 2 of the initiative and 57% increased their collaboration with other groups or organizations. Half of the working relationships between organizations within Urban Collaboratives were either “strong” or “very strong,” according to stakeholders, while working relationships within all Collaboratives strengthened during the second program year. All Rural Community Partners increased their evaluation skills; the areas of largest gains were seen in collecting data and using results to enhance existing programs. Eighty-three percent of Urban Collaboratives improved their evaluation skills most noticeably in collecting data, developing plans, and developing and identifying instruments. Eighty percent or more of the Rural Community Partners increased their access to volunteers, materials, technology, and technical assistance. Eighty-three percent of the Urban Collaboratives increased their access to funding or funding potential, and 67% increased their access to materials, technical assistance, or technology. Sixty-nine percent of regional and home office staff surveyed said that communication between Community Partners, regional office staff, and home office staff improved during Year 2 of the Web of Support; 85% of regional and home office staff reported that Community Partners showed increased awareness of national opportunities and efforts on behalf of children.
Youth Development	Over 5,600 youth encountered supportive adults through evaluated Save the Children sponsored programs or activities. Supportive adults included program staff, adult and teen volunteers, interns, tutors, teachers, and community elders. Of the 3,304 youth evaluated, 90% indicated that they had developed a positive relationship with a caring adult through their Save the Children activities. All of the youth who completed a Youth Summit survey felt confident in their ability to speak out in public on youth issues and 98% had a good understanding of how the media could help them get their message out. Participants in the internship program reported learning skills such as motivating peers and getting along with a wide range of people. All of the interns reported that the experience had helped them to “become a better leader.” Of the 4,370 youth evaluated, 73% exceeded or met expectations in one or more areas of cognitive, intellectual, social, or emotional growth.

A Profile of the Evaluation of Save the Children Web of Support Initiative

Profile Information