You are seeing this message because your web browser does not support basic web standards. Find out more about why this message is appearing and what you can do to make your experience on this site better.

www.HFRP.org

The Harvard Family Research Project separated from the Harvard Graduate School of Education to become the Global Family Research Project as of January 1, 2017. It is no longer affiliated with Harvard University.

Terms of Use ▼

Program Description

Overview The Save the Children Web of Support Initiative promotes quality out-of-school time programs for children ages 5–18 by focusing on three desired outcomes: constructive activities, caring adults, and safe places. Save the Children provides resources and assistance to community-based organizations, including school districts, to operate both school-site and community-site after school programs.
Start Date 1997
Scope national
Type after school, summer/vacation
Location urban, rural
Setting public schools, community-based organizations
Participants kindergarten through high school students
Number of Sites/Grantees over 100 Rural Community Partners and six Urban Collaboratives
Number Served approximately 7,000–9,000 youth
Components The Web of Support Initiative provides support in three areas: funding and resources, leadership training and development to improve program practices and program quality, and technical assistance and training in how to implement and use outcome evaluation as a framework for program improvement. Rural Community Partners are typically single local nonprofit agencies serving youth in rural areas. These Partners are located in rural areas in the Southwestern, Appalachian, and Eastern regions of the United States. Urban Collaboratives typically consist of two or more nonprofit organizations working together to serve at-risk youth in low-income urban environments. Each Urban Collaborative has a Facilitator who assists with the coordination of the Collaborative. Collaboratives are located in or near large metropolitan areas.

As part of the Web of Support initiative, Save the Children sought to develop the leadership capabilities of youth involved in the programs. Save the Children provided three different types of leadership development experiences for youth: the Save the Children Youth Summit, youth internships sponsored by Save the Children through Rural Community Partners and Urban Collaboratives (e.g., youth working as interns for the youth-serving nonprofit organizations), and other leadership opportunities offered in their communities by Partners and Collaboratives.
Funding Level $32 million (over three years)
Funding Sources Save the Children ($26 million over three years), the David and Lucile Packard Foundation ($3 million over three years), Denny's Family Restaurants (up to $1 million per year), and Irvine Foundation (Central Valley, California, expansion efforts)
Other The National Institute on Out-of-School Time (NIOST) provides training and technical assistance to Rural Community Partners, Urban Collaboratives, and Save the Children regional office staff to improve the quality of out-of-school time programs. Aguirre International provides evaluation training and technical assistance to Save the Children regional and home office staff, Rural Community Partners, and Urban Collaboratives on how to evaluate programs and activities. The evaluation training uses an empowerment evaluation approach. Professional consultants (coaches) provide Train-the-Trainer workshops, on-site assistance, and phone assistance to staff. Annual evaluation trainings are also held in each region.


Evaluation

Overview The Web of Support evaluation has two parts: (1) individual outcomes-based evaluations at the site level and (2) an initiative-wide evaluation.

In the first part of the evaluation, each Rural Community Partner and Urban Collaborative conducts individual outcomes-based evaluations annually, with technical assistance from Aguirre International. They are required to examine the three components of a comprehensive out-of-school time program of the Web of Support Initiative: constructive activities, caring adults, and safe places. In keeping with the empowerment model of evaluation, the data collection methods used by Rural Community Partners and Urban Collaboratives vary.

The purpose of the second part of the evaluation, the initiative-wide evaluation, is to examine three types of outcomes: child/youth outcomes, Rural Community Partner and Urban Collaborative outcomes, and Save the Children–U.S. programs outcomes. These evaluations are conducted annually, and the third-year evaluation looked at findings across the first three years in addition to that year's findings.
Evaluator Aguirre International
Evaluations Profiled 1997–98 Web of Support Initiative wide evaluation. Annual report.
1998–99 Web of Support Initiative wide evaluation. Annual report.
1999–2000 Web of Support Initiative wide evaluation. Annual report.
2000–2001 Web of Support Initiative wide evaluation. Annual report.
Evaluations Planned N/A
Report Availability Terao, K. L., Morell, L. C., Stevenson, C. L., & Sloane, K. J. (1999). 1997–98 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group.

Terao, K. L., Morell, L., & Stevenson, C. (2000). 1998–99 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group.

Terao, K. L., Morell, L., & Stevenson, C. (2001). 1999–2000 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group.

Terao, K. L., Morell, L., Stevenson, C., & Moulton, J. (2002). 2000–2001 Web of Support Initiative wide evaluation. Annual report. San Mateo, CA: The Aguirre Group.

Contacts

Evaluation Ken Terao
Evaluation Specialist
480 East 4th Avenue, Unit A
San Mateo, CA 94401-3349
Tel: 650-373-4900
Fax: 650-348-0260
Email: kterao@aiweb.com
Program Leigh Higgins
Save the Children Federation Inc. U.S.
54 Wilton Road
Westport, CT 06880
Tel: 203-221-4000
Fax: 203-221-4082
Email: lhiggins@savechildren.org
Profile Updated August 19, 2004

Evaluation 1: 1997–98 Web of Support Initiative Wide Evaluation: Annual Report


Evaluation Description

Evaluation Purpose To examine Year 1 of the initiative and the three types of outcomes related to the initiative: child and youth outcomes, Rural Community Partner and Urban Collaborative outcomes, and Save the Children–U.S. programs outcomes.
Evaluation Design Non-Experimental: Information about child and youth outcomes was obtained through reviews of evaluation reports produced by each Rural Community Partner and Urban Collaborative. Evaluators conducted additional data collection to examine Rural Community Partner and Urban Collaborative outcomes and Save the Children–U.S. Programs outcomes.

Over 3,000 rural youth and 500 urban youth participated in programs that were evaluated to determine children's perceptions of the caring adults involved in the programs and activities; of these youth, 1,319 rural youth and 171 urban youth were part of the evaluation.
Data Collection Methods Document Review: Data on child and youth outcomes were gathered through content analysis of Rural Community Partner and Urban Collaborative outcome evaluation reports. These evaluations were conducted between October 1997 and December 1998. Sites were given some latitude in terms of the content of these reports, but most looked at students' academic and developmental growth.

Surveys/Questionnaires: Youth involved in the youth leadership activities were asked to complete a survey to gather demographic data, information about their leadership skills, and information about their internship experiences and expectations.

Those participating in the National Leadership Collaborative Conference Training were also asked to complete a survey that measured participants' understanding of the conference's goals and the training presented, the roles of the training and technical assistance providers, and new ideas and strategies to help youth.

Surveys were administered to Rural Community Partners and Urban Collaborative members to gather information about the degree to which Rural Community Partners were collaborating, their skills in developing quality out-of-school time programming and internal evaluations, and the degree to which Partners and Collaborative members were communicating with community members and Save the Children.

Telephone surveys of Rural Community Partners and Urban Collaborative members collected information about the implementation of the Web of Support questions focused on demographics, the level, acquisition, and sharing of skills in developing and maintaining quality out-of-school time programming and evaluation, and the way in which Web of Support had affected their agency.

A telephone survey of Save the Children home and regional office staff measured the staff's impressions of the Leadership Conference (if they attended), the degree to which they and their staff understand how to provide training and technical assistance to Rural Partners and Urban Collaboratives, the effectiveness of National Institute on Out-of-School Time (NIOST) and Aguirre International's training methods, and their impressions of the Web of Support.
Data Collection Timeframe Data were collected between October 1997 and December 1998.


Findings:
Formative/Process Findings

Activity Implementation Partners provided tutoring, reading programs and activities, computer clubs, sports, recreation, creative arts, drama, cultural programs and activities, art programs and activities, other academic improvement programs, and mentoring programs.

Caring adults available for youth included program staff, adults and teen volunteers, tutors, youth interns, teachers, and seniors within communities being served by Save the Children.

Both urban and rural youth participated in the following programs and activities to enhance their leadership skills: the National Leadership Collaborative Conference II, internships sponsored by Save the Children through Rural Community Partners and Urban Collaboratives, and other leadership opportunities offered by these organizations.

Outside of Save the Children's national internship program, at least four Rural Community Partners and at least three Urban Collaboratives provided leadership programs and activities for youth that included youth councils/boards, community service opportunities, and community outreach programs.

Ninety-one percent of urban and rural youth interns helped plan or implement a project or activity.
Program Context/Infrastructure Eighteen Rural Community Partners reported that they made modifications or enhanced the maintenance of their facilities to improve the environment for youth participants. These included maintenance of adequate safety standards, improvement of facilities, modifications of activity areas to provide adequate space for activities, and enhancement or maintenance of outside environments.

Two Urban Collaboratives reported enhancing their facilities to provide youth participants with a safe place after school. Both reported recruiting or training volunteers or staff to provide supervision in programs and activities. Other changes included acquiring more space, developing a security system, and setting up more signs.
Systemic Infrastructure Fifty-four percent of pilot year Rural Community Partners and 47% of Year 1 Rural Community Partners stated that they felt comfortable or very comfortable with their evaluation skills, from developing evaluation plans to using the results to improve their programs.

Sixty-three percent of the Urban Collaboratives stated that they felt comfortable developing evaluation plans and conducting evaluations of their activities. One hundred percent of the Urban Collaboratives stated that they felt comfortable using evaluation results to enhance existing programs, while 88% felt comfortable using the results to develop new programs.

Of the Rural Community Partners participating in the Web of Support, 84% stated that they used Web of Support techniques or skills in other activities in their organization, 63% stated that they shared Web of Support techniques or skills with other Save the Children-affiliated organizations.

Of the Urban Collaboratives participating in the Web of Support, 82% stated that they used Web of Support techniques or skills with other activities in their organization; 47% stated they shared Web of Support techniques or skills with other Save the Children organizations; 88% shared Web of Support techniques or skills with organizations not affiliated with Save the Children.

Rural Community Partners and Urban Collaboratives were offered both formal and informal training and technical support (T/TA) from NIOST, Aguirre International, and the Save the Children regional and home office staff. Formal T/TA included participation in workshops, usually lasting one to three days, and site visits. Informal T/TA included phone assistance and sending material to the Rural Community Partners and Urban Collaboratives. The content areas T/TA provided consisted of program improvement, evaluation, leadership development, community development, collaboration, and mobilizing resources. During their end-of-year interviews, both the Rural Community Partners and Urban Collaboratives described the T/TA they received as very helpful.


Summative/Outcome Findings

Academic Of the 1,421 rural youth evaluated by Partners in academic programs or activities, 75% demonstrated positive academic outcomes, such as increased test scores, grades, or grade point averages, or improvements on pre/post surveys.

Of the 707 urban youth evaluated by Partners in academic programs and activities, 62% demonstrated positive academic outcomes such as improved school grades or attitude toward school, increased reading skills, or increased communication and decision-making skills.
Community Development Seventy percent of Rural Community Partner respondents reported that their local communities were more aware and supportive of Save the Children's efforts than they were at the beginning of the evaluation period.

Ninety-four percent of the Urban Collaborative respondents stated that their local communities were more aware and supportive of Save the Children's efforts than they were at the beginning of the evaluation period.
Systemic Sixty-four percent of Rural Community Partners reported collaborating with other Partners “somewhat” or more often by the end of Year 1.

Ninety-two percent of Urban Collaborative members reported that they were collaborating “quite a bit” or “a lot” with other members of their Collaborative by the end of Year 1, as compared to 62% reporting that they were collaborating “quite a bit” or “a lot” during the first three months of joining the Web of Support. Rural Community Partners and Urban Collaboratives both reported sharing ideas and resources.

All of the pilot year Rural Community Partners stated that they increased their skills in using the evaluation results to develop new programs or enhance existing programs.

Of the Rural Community Partners, 89% reported an increase in materials, 86% reported an increase in technical assistance, and 82% reported an increase in volunteers since they began participating in the Web of Support.

All of the Urban Collaboratives reported an increase in volunteers and materials, and 88% reported an increase in funding or technical assistance since they began participating in the Web of Support.

Eighty-nine percent of Rural Community Partner staff stated that being a part of the Web of Support helped them improve the quality of service they provide to youth.

Ninety-four percent of the Urban Collaboratives stated that the Web of Support helped them to improve services.

Eighty percent of the Save the Children regional and home office staff indicated that communication improved between Rural Community Partners, Urban Collaboratives, regional offices, and the home office.

All of the Save the Children regional and home office staff reported that there was an increase in awareness of national opportunities on behalf of children.

Fifty percent of the Save the Children regional office staff stated that they had “to a great extent” gained the skills to provide technical assistance in program improvement to Rural Community Partners.

Seventy percent of the Save the Children regional and home office staff stated that they had “to a great extent” gained the skills to provide technical assistance in evaluation to Rural Community Partners and Urban Collaboratives.
Workforce Development Of the 75 youth evaluated in the vocational/craft skills training program, 76% developed knowledge of arts and skills in the use of craft tools and techniques.
Youth Development Of the 1,319 rural youth evaluated, 78% indicated they had a caring adult in their life as a result of participating in a Rural Community Partner program or activity. Of the 171 urban youth evaluated, 84% indicated that they had a caring adult in their life as a result of their participation in an Urban Collaborative activity.

Of the 2,244 rural youth evaluated in social skills enhancement programs or activities, 70% showed positive behavior or an improvement in behavior with other children or adults.

Both urban and rural youth reported that they learned new skills at the Leadership Conference such as leadership, communication, planning and organizing, prioritizing and time management, and teamwork skills.

Both urban and rural youth reported that they learned skills through their internship experiences that included responsibility, teamwork, communication, how to work with children, and job skills.

The results of a pre/post survey indicated that leadership skills increased in 29%–35% of the urban and rural youth interns in one or more areas, including ability to motivate others, communication, general leadership, conflict resolution, and organizing/planning.

Evaluation 2: 1998–99 Web of Support Initiative Wide Evaluation: Annual Report


Evaluation Description

Evaluation Purpose To examine Year 2 of the initiative and three types of outcomes related to the initiative: child and youth outcomes, Rural Community Partner and Urban Collaborative outcomes, and Save the Children–U.S. Programs outcomes.
Evaluation Design Non-Experimental: Information about child and youth outcomes was obtained through review of evaluation reports produced by each Rural Community Partner and Urban Collaborative. Evaluators conducted additional data collection to examine Rural Community Partner and Urban Collaborative outcomes and Save the Children–U.S. Programs outcomes.
Data Collection Methods Document Review: Data on child and youth outcomes were gathered through content analysis of Rural Community Partner and Urban Collaborative outcome evaluation reports. These evaluations were conducted between October 1998 and September 1999.

Surveys/Questionnaires: Youth involved in the youth leadership activities were asked to complete a survey to gather information about their knowledge and leadership skills and about internship experiences and expectations.

Surveys were administered to Rural Community Partner members to gather information about the degree to which Rural Community Partners were collaborating, their skills in developing quality out-of-school time programming and internal evaluations, and the level of resources they were able to access because of their involvement in the Web of Support.

Surveys of Urban Collaborative members gathered information about the effect the Web of Support had had on youth, their individual organizations, and local communities, the way in which internal evaluation results had been used, the value of having a facilitator, and the elements that worked and did not work well in their Collaborative.

Surveys of Urban Collaborative Facilitators gathered information on the level of collaboration between partner organizations; the resources Collaborative Partners had increased access to; the effect the Web of Support had on youth, the Collaborative, and local communities; and the extent to which organizations in their Collaborative increased their evaluation skills.

Surveys of Save the Children home and regional office staff measured their understanding of how to provide training and technical assistance to Rural Partners and Urban Collaboratives, the effectiveness of NIOST and Aguirre International's training methods, the technical assistance they and their Partners need in the coming year, and the impact the Web of Support has had on Save the Children.
Data Collection Timeframe Data were collected between August and November 1999.


Findings:
Formative/Process Findings

Activity Implementation Partners provided tutoring, recreation, computer training, leadership development, cultural enrichment, and other innovative programs designed to meet the needs of children in their community. Leadership development activities included an internship program, mentoring, facilitating activities for children, attending trainings and workshops, and delivering presentations at community meetings. Internships provided youth with the opportunity to be a part of youth groups working for change or to plan and participate in projects that focused on community service and outreach efforts.
Parent/Community Involvement All Save the Children staff indicated their Community Partners demonstrated leadership skills by working with local community leaders to improve services for youth.

Urban Collaboratives demonstrated leadership qualities by sharing staff, volunteers, interns, and office space within their Collaboratives.

Rural Community Partners worked with other local organizations to provide academic, cultural, recreational, and other services in their after school programs.
Program Context/Infrastructure Thirty-six Rural Community Partners and three Urban Collaboratives reported that they completed structural or landscape maintenance upgrades at program sites so that youth had a safe place for programs and activities.

Partners reported that they provided youth with space for activities, comfortable atmospheres, safety precautions, and healthy snacks.
Systemic Infrastructure Rural Community Partners received training and technical assistance in program improvement, community and leadership development, and mobilizing resources from NIOST.

Urban Collaboratives received training and technical assistance in program improvement, youth leadership development, and collaboration from the home office.

Most Rural Community Partners and Urban Collaboratives surveyed reported that the training they received from NIOST and Aguirre International was “helpful” or “very helpful.”

Eighty-two percent of regional and home office staff interviewed reported that they gained the skills to provide technical assistance to Community Partners (rural and urban) on improving their programs.

Ninety-one percent of the regional and home office staff said that they gained the skills to provide technical assistance in helping Community Partners with their evaluation efforts.

Eighty-two percent of the regional and home office staff surveyed reported that the Train-the-Trainers model was “very effective” in helping them to become coaches for Rural Community Partners or Urban Collaboratives.


Summative/Outcome Findings

Academic Of the 1,080 students evaluated in tutoring/homework assistance programs, 80% maintained high standards or showed improvements in one or more of the following: grades, homework completion, study habits, or other measures of academic success.

Of the 1,987 youth evaluated in reading programs, 60% exceeded or met expectations in reading ability, comprehension, or interest.
Community Development Ninety-two percent of Rural Community Partners reported that their community was more aware that Save the Children provided services in their communities during the past year.

Sixty-seven percent of Urban Collaborative facilitators reported that their local community was more aware of Save the Children's contributions to their neighborhoods during the past year.
Systemic Seventy-four percent of Rural Community Partners reported collaborating with other Save the Children Community Partners during Year 2 of the initiative and 57% increased their collaboration with other groups or organizations.

Half of the working relationships between organizations within Urban Collaboratives were either “strong” or “very strong,” according to stakeholders, while working relationships within all Collaboratives strengthened during the second program year.

All Rural Community Partners increased their evaluation skills; the areas of largest gains were seen in collecting data and using results to enhance existing programs.

Eighty-three percent of Urban Collaboratives improved their evaluation skills most noticeably in collecting data, developing plans, and developing and identifying instruments.

Eighty percent or more of the Rural Community Partners increased their access to volunteers, materials, technology, and technical assistance.

Eighty-three percent of the Urban Collaboratives increased their access to funding or funding potential, and 67% increased their access to materials, technical assistance, or technology.

Sixty-nine percent of regional and home office staff surveyed said that communication between Community Partners, regional office staff, and home office staff improved during Year 2 of the Web of Support; 85% of regional and home office staff reported that Community Partners showed increased awareness of national opportunities and efforts on behalf of children.
Youth Development Over 5,600 youth encountered supportive adults through evaluated Save the Children sponsored programs or activities. Supportive adults included program staff, adult and teen volunteers, interns, tutors, teachers, and community elders.

Of the 3,304 youth evaluated, 90% indicated that they had developed a positive relationship with a caring adult through their Save the Children activities.

All of the youth who completed a Youth Summit survey felt confident in their ability to speak out in public on youth issues and 98% had a good understanding of how the media could help them get their message out.

Participants in the internship program reported learning skills such as motivating peers and getting along with a wide range of people. All of the interns reported that the experience had helped them to “become a better leader.”

Of the 4,370 youth evaluated, 73% exceeded or met expectations in one or more areas of cognitive, intellectual, social, or emotional growth.

Evaluation 3: 1999–2000 Web of Support Initiative Wide Evaluation: Annual Report


Evaluation Description

Evaluation Purpose To examine Year 3 of the initiative and three types of outcomes related to the initiative: child and youth outcomes, Rural Community Partner and Urban Collaborative outcomes, and Save the Children–U.S. Programs outcomes. The findings reflect both the third year, as well as the past three years overall.
Evaluation Design Non-Experimental: Information about child and youth outcomes was obtained through review of evaluation reports produced by 58 Rural Community Partners and eight Urban Collaboratives. Evaluators conducted additional data collection to examine Rural Community Partner and Urban Collaborative outcomes and Save the Children–U.S. Programs outcomes.
Data Collection Methods Document Review: Data on child and youth outcomes were gathered through content analysis of Rural Community Partner and Urban Collaborative outcome evaluation reports. These evaluations were conducted between October 1999 and September 2000.

Observation: Informal observations were conducted during site visits and trainings.

Surveys/Questionnaires: Youth involved in the youth leadership activities were asked to complete a survey to gather information about their knowledge and leadership skills and about internship experiences and expectations.

Surveys were administered to Rural Community Partner members to gather information about the degree to which Rural Community Partners were collaborating, their skills in developing quality out-of-school time programming and internal evaluations, and the level of resources they were able to access because of their involvement in the Web of Support.

Surveys of Urban Collaborative members gathered information about the effect the Web of Support had had on youth, their individual organizations, and local communities; the way in which internal evaluation results had been used; the value of having a facilitator; and the elements that worked and did not work well in their Collaborative.

Surveys of Urban Collaborative Facilitators gathered information on the level of collaboration between partner organizations; the resources Collaborative Partners had increased access to due to the Web of Support Initiative; the effect the Web of Support had on youth, the Collaborative, and local communities; and the extent to which organizations in their Collaborative increased their evaluation skills.

Surveys of Save the Children home and regional office staff measured their understanding of how to provide training and technical assistance to Rural Partners and Urban Collaboratives, the effectiveness of NIOST and Aguirre International's training methods, the technical assistance they and their Partners need in the coming year, and the impact the Web of Support has had on Save the Children.
Data Collection Timeframe Data were collected over the three years of the initiative, 1997–2000. Findings focus on the third year (1999–2000), with some outcomes summarizing changes over the three-year period.


Findings:
Formative/Process Findings

Activity Implementation Save the Children Partners and Collaboratives provided tutoring, computer training, leadership development, community service, education in prevention, cultural enrichment, and other programs designed to meet the needs of youth in their community.

Save the Children Partners and Collaboratives provided programs that promoted social or emotional growth, such as conflict resolution and communication, leadership development activities, cultural enrichment programs, prevention education, and other innovative programs designed to meet the needs of children in their community.

Rural Community Partners and Urban Collaboratives provided leadership opportunities for youth, including an internship program, youth councils and youth groups, workshops, conferences, trainings, tutoring and mentoring of younger children, and community service projects.

Internships provided youth with the opportunity to plan and implement projects that focused on community events, such as cultural festivals, parades, and “youth days,” performing arts activities, like plays and poetry nights, and community service projects, including food distribution and environmental beautification projects.
Parent/Community Involvement Eighty-five percent of Save the Children staff members indicated that their Community Partners demonstrated leadership skills by working with local community leaders to improve services for children.
Systemic Infrastructure Rural Community Partners received training and technical assistance in program improvement, community and leadership development, and resource mobilization from NIOST or other program development specialists.

Urban Collaboratives received training and technical assistance in program improvement, youth leadership development, and collaboration from Save the Children.

Both Rural Community Partners and Urban Collaboratives received training and technical assistance in evaluation from Aguirre International, in collaboration with Save the Children's home office (in urban areas) and regional office staff (in rural areas). Most Rural Community Partners and Urban Collaboratives surveyed reported that the training they had received was “helpful” or “very helpful.”


Summative/Outcome Findings

Academic Of the 4,057 youth evaluated, 83% exceeded or achieved the desired result in one or more areas of cognitive/intellectual growth. (The desired result varied by each Partner's evaluation criteria.)

Of the 1,412 students attending tutoring/homework assistance programs who were evaluated, 84% maintained high standards or showed improvement in one or more of the following: grades, homework completion, study habits, or other measures of academic success.

Of the 907 youth evaluated in reading programs, 88% exceeded or achieved the desired result in reading ability, comprehension, or interest. (The desired result varied by each Partner's evaluation criteria.)
Community Development All of the Rural Community Partners and all of the Urban Collaborative facilitators reported that their community was aware that Save the Children provides services in their communities; 74% of Rural Community Partners and all Urban Collaborative facilitators said that this awareness had increased since their involvement with the Web of Support.
Systemic Ninety-four percent of the Rural Community Partners reported collaborating with other Save the Children Community Partners, and 41% of those indicated that their collaborations have increased over the course of the initiative. The various ways in which they collaborated included regular communication, joint planning, joint decision making, resource sharing, or joint program implementation.

Among the seven Urban Collaboratives, working relationships between organizations in 57% of the Collaboratives were rated by stakeholders as either “strong” or “very strong” and 57% reported that they had strengthened their working relationships either “somewhat” or “a lot.”

Among Rural Community Partners, approximately one-half reported that they had improved their evaluation skills and practices over the course of the initiative “quite a bit,” and an additional one-quarter had improved skills “a lot.” The evaluation skills and practices improved included: developing evaluation plans, developing or identifying instruments, collecting and analyzing data, and writing evaluation reports.
Of the six Urban Facilitators reporting on various evaluation skills and practices that Urban Collaboratives had acquired, 17% to 50% reported that their Collaboratives had improved “quite a bit” and another 20% reported that they had improved “a lot.”

Rural Community Partners had access to additional resources as a result of participation in Web of Support. In particular, resources included technical assistance, materials, and technology.

Urban Collaboratives had access to additional resources such as materials, technology, and facilities or space due to Web of Support.

Eighty-six percent of regional and home office staff surveyed said that communication between Community Partners, regional office staff, and home office staff improved.

Ninety-three percent of regional and home office staff reported that Community Partners showed increased awareness of national opportunities and efforts on behalf of children.

Ninety-three percent of regional and home office staff interviewed reported that they gained the skills to provide technical assistance to Community Partners (rural and urban) in improving their programs.

All of the regional and home office staff said that they gained the skills to provide technical assistance in helping Community Partners with their evaluation efforts.

Sixty-two percent of the regional and home office staff surveyed reported that the Train-the-Trainers model was “very effective” in helping them to become evaluation coaches for the Rural Community Partners or Urban Collaboratives.
Youth Development Of the 5,208 youth evaluated, 86% exceeded or achieved the desired result in one or more areas of social or emotional growth. (The desired result varied by each Partner's evaluation criteria.)

Of the 2,066 youth evaluated in programs designed to develop social skills, 85% showed improvement or maintained good behavior.

Of the 704 youth evaluated in cultural enrichment programs, 81% increased their interest, appreciation, or pride in their cultures.

Ninety-six percent of the youth interns who attended the Youth Voice 2000 Millennium Leaders Summit reported that Save the Children had given them valuable leadership skills and that they had learned new ideas.

Over 95% of the youth interns enhanced their communication skills and their ability to get along with a wide range of people, to collaborate, to take responsibility, and to lead.
Evaluation 4: 2000–2001 Web of Support Initiative Wide Evaluation: Annual Report

Draft Version: This profile has not yet received feedback from evaluators or program staff.
Evaluation Description
Evaluation Purpose To examine Year 4 of the initiative and four types of outcomes related to the initiative: child and youth outcomes, Rural Community Partner and Urban Collaborative outcomes, Save the Children–U.S. Programs outcomes, and community outcomes.
Evaluation Design Non-Experimental: Save the Children Community Partners evaluated approximately 70 constructive activities designed to enhance cognitive/intellectual growth and about 60 activities meant to facilitate social/emotional or physical health development. Data were collected from 58 Rural Community Partners and seven Urban Community Partners. Information about child and youth outcomes was obtained by evaluators through review of evaluation reports produced by the Partners. Evaluators conducted additional data collection to examine Partners' outcomes and Save the Children–U.S. Programs outcomes.
Data Collection Methods Document Review: The initiative-wide evaluation examined approximately 140 outcome evaluation reports compiled by individual programs. For the 2000–2001 year, Web of Support focused on evaluation of cognitive/intellectual outcomes and a second outcome area of each Partner's choice (e.g., social/emotional, health, additional cognitive/intellectual, etc.). Most Partners focused their evaluation reports on academic assistance (tutoring and homework help) or literacy programs. Several Partners focused on computer-training programs or activities promoting progress in social skills (e.g., recreation programs).

Interviews/Focus Groups: Telephone interviews were conducted with Save the Children regional and home office staff to measure the degree to which they and designated staff had learned how to provide training and technical assistance in outcome evaluation and quality program improvement, the effectiveness of Aguirre International's training and technical assistance in evaluation, and the extent to which Partners had been able to use the evaluation process and results to improve their services.

Observation: Observations of programs were made during site visits and trainings to ensure the quality of the data collected by individual programs.

Surveys/Questionnaires: Surveys were developed and administered to examine outcomes for four identified stakeholder groups: youth interns, Rural and Urban Community Partners, Save the Children Federation, Inc. – US programs, and communities.

Surveys were administered to youth interns who attended the National Youth Summit or participated in Save the Children's internship program. These were youth selected by individual Partners to participate in the internship program.

In addition, surveys were administered to all Community Partners to examine how they were using evaluation and their level of skills in developing quality out-of-school time programming and internal evaluations. These surveys also asked about their programs' collaborative efforts with other stakeholders in the community and about their youth leadership development opportunities.
Data Collection Timeframe Data were collected between October 2000 and December 2001.

Findings:
Formative/Process Findings
Activity Implementation Partners provided tutoring, literacy enhancement, computer training, community service, education in prevention, recreation and teambuilding activities, cultural enrichment programs, leadership development opportunities, and other innovative programs designed to meet the needs of youth in their communities.

One site in Zuni, New Mexico, implemented the Zuni Pueblo's Native-American Authors Project for students aged 14–18, in which six Native-American authors shared knowledge with youth on writing and publishing. After attending lectures and workshops, the students had the opportunity to write an essay and submit it to a Language Art Team, who critiqued the students for content, punctuation, spelling, and grammar. Of the 182 students evaluated, 84% published one or more original stories in a student publication.

Ninety-two percent of the youth interns surveyed had opportunities to plan and implement community projects or activities, such as festivals and cultural events, environmental beautification projects, food and clothing distribution to needy families, and literacy programs.

Save the Children held a 4-day Youth Summit in April 2001. The 4-day Summit gave participants a chance to meet other youth from around the country, to better understand Save the Children's mission, to learn about the legislative process and how to work with the media, and to help them prepare for their internships. The overall purpose of the Summit was to challenge and motivate youth to take an advocacy role in support of their communities. Youth attended workshops where they were given practical, step-by-step instructions in how to define and set achievable goals, research topics using various accessible, reliable sources, and speak to people in positions of power. Youth chose issues and worked in groups to prepare for the visit to Capitol Hill on the fourth day of the Summit, where they made presentations to representatives in Congress or their staffs on issues that were important to them.

Approximately 93 youth participated in Save the Children's national internship program during the 2000–2001 school year or summer.

When asked about the responsibilities of their internships, over 90% of the respondents reported that being a positive role model or supervising and working with younger children were some of their duties. Eighty-one percent of the interns reported that their responsibilities were planning and organizing activities. Between 55% and 70% indicated that they were responsible for mentoring, managing projects or activities, assisting staff with implementing activities, tutoring, maintaining the facility (e.g., cleaning, repairs, general upkeep), motivating peers, training other youth, doing community service work, and running errands or assisting staff.
Systemic Infrastructure Ninety-eight percent of the Community Partners reported collaborating with other groups and organizations since October 2000 to provide services to children. Eighty-five percent of the 60 responding Partners had collaborated with elementary and secondary schools and school districts. About one-third (22) had worked with community organizations, and another third with state and county agencies. Roughly 18% had worked with churches, and 10–13% with social service organizations, local youth organizations, or local colleges. Some worked with senior citizen organizations, housing authorities, summer school and after school programs, hospitals, Head Start, and local TV stations.

When asked about the types and degree of collaboration among Partners since October 2000, the following percentages of Partners reported “quite a bit” of each type of collaboration: regular communication – 50%, shared resources – 40%, planned together – 34%, and collaborated in program implementation – 29%. Other collaborative efforts reported by Community Partners included shared meeting places and recruitment of additional volunteers.

Most of the Rural and Urban Community Partners surveyed reported that the training they had received in program improvement, evaluation, collaboration, and youth leadership development was “helpful” or “very helpful.” Partners particularly valued the “hands-on” learning approach.

The telephone survey of nine Save the Children home and area office staff members revealed that all respondents provided training and technical assistance (T/TA) to their Community Partners in program development and in evaluation, and eight of the nine provided T/TA in collaboration and youth leadership.

Community Partners reported being “quite” comfortable in conducting evaluation tasks; three-quarters or more reported being comfortable using results to enhance existing programs and to develop new programs.

Most Community Partners found the evaluation process useful, and approximately two-thirds reported having made use of evaluation findings to modify their programs.

Most home and area office staff reported that Partners utilized evaluation for fundraising, resource development, program improvement, and planning.

Seven of the nine home and area office staff members surveyed reported that they felt their evaluation skills were very good.


Summative/Outcome Findings

Academic Of the 5,358 evaluated youth, 71% achieved or exceeded the desired result in one or more areas of cognitive/intellectual growth. (The desired result varied by each Partner's evaluation criteria.)

Seventy-one percent of the 1,741 students evaluated in a tutoring/homework assistance program met or exceeded the program's level of success by maintaining high standards or showing improvement in one or more of the following: grades, homework completion, study habits, or other measures of academic success.

Seventy-two percent of the 2,706 students evaluated in literacy programs achieved or exceeded the desired result in reading ability, comprehension, or interest. (The desired result varied by each Partner's evaluation criteria.)
Community Development Of the 50 Community Partners who responded, either nine or ten reported that their participation in the Web of Support had affected their community in each of the following ways: community participation and comfort with their program had increased, they had been able to provide a safety net for local children and families, they had been able to offer resources and activities for the community.

Some Partners reported that being a part of the Web of Support helped the community to work together to define needs and solve problems, and to see the positive effects of their programs. Other respondents reported that the community gained access to computer training and had improved family practices (i.e., in health care and child care).

Three Partners gave neutral or negative responses about the initiative's effect on their community. One reported that there had been no effect, another noted that the Web of Support had not helped them to provide activities for adults who needed services, and one was still working to implement the Web of Support program.
Systemic Of the 53 Partners who responded to a question about improvements in quality, 30% (16) said that they were able to provide youth with more resources, such as books, meals, supplies, and computer use because of their involvement with the Web of Support.

Over 70% of the Partners reported that they had “quite a bit” or “a lot” of understanding of how to improve an out-of-school time program.

Communication among Save the Children staff and Community Partners generally improved: Over two-thirds felt that communication had improved either “some” or “a lot.”

In regards to program improvement at Community Partners sites, seven of the nine respondents indicated that they had acquired strong program development skills, including learning how to help Partners increase resources and enhance program capacity.
Youth Development Of 4,893 youth evaluated in areas other than cognitive/intellectual activities, 78% exceeded or achieved the desired result in one or more areas of cognitive/intellectual growth, social/emotional development, or health-related development.

Seventy-three percent of the 1,689 students evaluated in programs designed to develop social skills showed improved behavior or maintained their good behavior.

Eighty-four percent of the 1,245 students evaluated in cultural enrichment programs increased their knowledge of, interest in, appreciation for, or pride in their cultures.

Ninety-five percent of those who attended the Youth Summit reported learning about other cultures, and 99% reported learning about what other youth are doing in their communities.

When asked what they had learned about promoting a cause or lobbying for social change, 47 of 80 Youth Summit attendees responded that they had learned to stand up for what they believed without feeling intimidated, to approach different people and communicate their ideas confidently, to conduct presentations, and to prepare through careful research and education.

The proportion of Youth Summit attendees who felt comfortable making public presentations rose from just under 60% before the Summit to just over 90% afterward.

Some Summit attendees reported that they learned about action they could take to resolve specific issues important to their communities such as education in substance abuse or the need to acquire recreation centers in their communities.

The majority of Summit survey respondents (94%) reported that they learned how to find information on social issues.

All of the interns surveyed believed they learned to take responsibility, and 99% reported that they became better leaders. They also learned to communicate better, collaborate with others, work in teams, and supervise young children.

© 2016 Presidents and Fellows of Harvard College
Published by Harvard Family Research Project